Pith.

Machine Learning · arXiv:2611.16409 v1 · submitted 12 Nov 2026 · 92 references

Experience-Handoff Reinforcement Learning for Recursive Self-Improvement

Doyoung Kim, Byeongguk Jeon, Kailin Zhang, Sherry Yang

The paper says

“It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs).”

more of the abstract

To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 13 Nov 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Single training run per method limits statistical robustness despite careful design. This line is wrong

2 major objections · 7 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 92 references say.

The paper says, at Section 4, Table 2, Figure 4, and Appendix H

The review did not quote this passage.

A classifier read this sentence of the paper as acknowledging the concern (probability 0.72):

“Measuring that variance directly, and establishing with additional training runs when it pays off, is left to future work. 17 Experience-Handoff Reinforcement Learning for Recursive Self-Improvement C ROLEPROMPTS The listings below are the actor, reflector, and curator prompts used for training on each benchmark.”

Pith says: Objection 1 of 2 · major

Only one training run is reported per method. The paper acknowledges this in Appendix H ("Our evidence concerns… one training run per method"), but the central claim that ACE-RL outperforms all baselines is directly supported by this single run. Error bars over held-out tasks address within-run variability but not seed-level variance.

more of objection 1

The authors should either provide additional training runs (e.g., 3 seeds for one benchmark) to demonstrate stability, or add a prominent caveat in the main text that the results are from a single training run and may be sensitive to seed. This is load-bearing because the ranking could change with different random initializations, and the current presentation overstates the strength of the evidence.

The paper says, at Appendix D.1 and Section 3.2

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The derivation of the gradient estimator is careful, but the implementation uses several approximations: a single evaluation per candidate playbook (L=1), mixing seed and candidate rollouts in the same actor group (different inputs (q,P) vs (q,P')), and group filtering that drops constant-reward groups.

more of objection 2

These approximations are not guaranteed to preserve the expected gradient direction. While Appendix D.1 discusses these issues, the main text presents the method as a straightforward application of GRPO. The authors should add a brief statement in Section 3.2 that the update is a sparse, variance-reduced approximation to the full-tree estimator, and reference the appendix for details.

7 minor objections

The paper says, at Introduction (Section 1)

Pith says: minor objection 1 of 7

The term 'Experience-Handoff RL' is introduced but not formally defined until Section 2.2. Consider defining it earlier, after the first paragraph, to help the reader.

The paper says, at Figure 3

Pith says: minor objection 2 of 7

The font size in the figure is small; consider enlarging labels for readability, especially the role-group annotations.

show 5 more

The paper says, at Table 1

Pith says: minor objection 3 of 7

The 'Advantage' column uses abbreviations ('group', 'critic (GAE)') that could be expanded ('group-relative', 'GAE from learned critic') for clarity.

The paper says, at Section 3.1

Pith says: minor objection 4 of 7

The claim that 'the final playbook is verifiable' relies on a single rollout (Eq. 6). While group normalization reduces variance, the paper should note explicitly that the verifier reward is a noisy estimate, not a ground-truth value, and that noise is handled by the per-role groups.

The paper says, at Section 5.3, Figure 7

Pith says: minor objection 5 of 7

The paper says 'the gain does not fade with more attempts', but on Mastermind the benefit of applying edits fades after K=3 and from K=5 onward the discarded run is slightly ahead. This should be stated in the text rather than only visible in the figure.

The paper says, at Appendix B.1

Pith says: minor objection 6 of 7

The variance decomposition in Eq. (9) is insightful; consider citing it briefly in the main paper (e.g., after Section 3.2) to motivate the per-role grouping.

The paper says, at Section 6 (Conclusion)

Pith says: minor objection 7 of 7

The conclusion restates results without mentioning the single-run limitation. Even a short sentence (e.g., 'Results are based on one training run per method; see Appendix H for discussion') would be appropriate.

What would settle it

Run the main comparison of Table 2 (ACE-RL vs. GRPO vs. PPO vs. ERL vs. Delayed-joint PPO) on at least one benchmark (e.g., Mastermind or MBPP) with 5 independent random seeds per method, keeping all other hyperparameters fixed. Report the mean and standard deviation of test reward (RSI@3 for ACE-RL, pass@3 for GRPO) across seeds. If ACE-RL's mean is above GRPO's by at least 0.05 with non-overlapping standard deviations, the claim is robust; otherwise the advantage may be seed-dependent.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 13 Nov 2026

  1. Reader holds up, with conditions confidence high
  2. Skeptic partly agrees 25 s · 48,268 tokens read, 2,786 written
  3. Editor wrote the verdict above 13 s · 49,052 tokens read, 1,377 written
  4. Referee minor revision: 2 major objections, 7 minor 24 s · 48,524 tokens read, 2,740 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This paper introduces a real advance: training all three roles (actor, reflector, curator) in a harness using downstream rollout rewards and per-role comparison groups. The central claim—that ACE-RL outperforms actor-only RL, ERL, and chain-level credit methods—holds in the reported experiments. The experimental design is careful: matched compute, a control that discards edits, evaluation beyond the training horizon, and multiple baselines including a critic-based method that collapses. The paper is well written and transparent about its limitations.

The soft spot is the one you flagged. The authors ran one training seed per method. The error bars they report are over held-out tasks within that single run, not over training seeds. The stress-test note is right: with stochasticity in initialization, rollout sampling, group composition, and playbook resets, the observed advantage might not replicate under another seed. The authors acknowledge this in Appendix H, but it weakens the headline claim. A second concern: the method depends on a single rollout to score each candidate playbook. The paper says this works because the final playbook is verifiable, but in stochastic environments that signal can be noisy. The per-role groups mitigate variance, but they don't eliminate it.

The method also adds generation cost (reflector and curator calls) even though update FLOPs are matched. The code is promised but not yet released. The experiments are on models up to 4B parameters; scaling to frontier models is open.

This paper is for researchers in agentic RL and harness-based self-improvement. It deserves a serious referee and conditional acceptance. I would not desk reject it. My recommendation: ask for a second training seed as a robustness check, and release the code before final acceptance.

The strongest objection

The paper reports only one training run per method (Section 4, Table 2, Figure 4). All error bars are standard errors over held-out tasks within that single run, not over training seeds. The training process involves stochasticity in policy initialization, rollout sampling, group composition, and playbook-reset timing. Without multiple independent training runs, it is possible that ACE-RL's advantage over baselines (e.g., GRPO) is partially driven by a favourable seed. This concern is acknowledged in Appendix H ('Our evidence concerns … one training run per method'), but the central claim that ACE-RL outperforms all baselines at equal compute would be weakened if another seed yielded smaller or reversed gains. The paired comparison over tasks within a single run does not address seed-level variance.

The reader's verdict is CONDITIONAL, with the primary requested condition being code release and 'ideally additional seeds.' This concern is the single most load-bearing: the paper's headline empirical claim (ACE-RL outperforms baselines) rests on comparisons from a single training run per method. While the paper acknowledges this limitation, it does not resolve it, and the reader's condition already targets this. My concern aligns with the reader's call for additional seeds but goes further in highlighting that seed-level variance could affect the central conclusion. I recommend UNCHANGED because the reader's condition already accounts for this risk. The concern is not that the paper is wrong, but that the evidence is incomplete without multi-seed verification.

The referee's report

If the results hold, ACE-RL provides a practical method for training agents to improve their own experience handoffs, combining outcome-based RL with rich textual artifacts. The experimental design is careful: matched training compute, multiple baselines, a control (edits-discarded), and evaluation beyond the training horizon. The method is grounded in a verifiable reward for handoff artifacts, which avoids learned reward models. However, the central empirical claim rests on a single training run per method, which weakens the statistical robustness of the conclusions across random seeds.

Pith's reading of the paper (a machine's summary, not the authors')

Large language models trained with reinforcement learning for agentic tasks typically compress each trial into a scalar reward, discarding the rich environment feedback (error logs, test failures) that the agent experienced. Harnesses like ACE preserve that feedback as text artifacts (reflections, playbooks) that future attempts can read, but the model never learns to write better artifacts—the handoff is frozen. This paper introduces experience-handoff RL, in which a single model is trained from task outcomes alone to act, to reflect on its trajectory, and to curate a task-local playbook that improves future attempts. The method, ACE-RL, scores each candidate playbook by executing a fresh rollout with it and normalizes rewards within per-role comparison groups. Across four benchmarks (Mastermind, MBPP, AppWorld, MiniSWEBench), ACE-RL achieves higher held-out reward than actor-only RL, end-to-end chain training with a learned critic, and the common practice of running an actor-only checkpoint inside the same harness.

What carries the argument. The fixed Agentic Context Engineering (ACE) harness with three roles (actor, reflector, curator), the verifiable playbook reward (each candidate playbook P' is scored by a fresh rollout R_q(τ̃)), and per-role GRPO-style comparison groups (actions compared with other actor rollouts of the same task, reflections compared with other reflections of the same seed trajectory, edits compared with other edits of the same reflection). These allow joint optimization of all roles with one clipped surrogate update, without a learned critic.

Core claim. The paper claims that simultaneously training the actor, reflector, and curator roles of a fixed harness (ACE) from verifiable downstream rewards produces a model whose playbook edits actually help future attempts, while training only the actor does not. The key enablers are (i) a verifiable reward for each handoff artifact obtained by executing the candidate playbook in a fresh rollout, and (ii) per-role comparison groups that give each output (action, reflection, edit) an advantage normalized within its own group under a shared input, rather than sharing a chain’s final reward. On all four benchmarks the ACE-RL checkpoint beats GRPO, ERL, delayed-joint PPO, and the GRPO checkpoint run inside the ACE harness. The gain persists beyond the training horizon (F=3) up to K=8 attempts, and an edits-discarded control shows the playbook contributes beyond actor improvement alone.

The premise it rests on. The final playbook is verifiable: a single downstream rollout of the candidate playbook yields a useful reward signal for crediting the reflections and edits that produced it.

If the paper is right (Pith's inferences, not the authors')

Agentic RL pipelines should optimize not only the actor's actions but also the model's ability to write useful experience (reflections, playbooks) for future attempts.

The per-role comparison group design can be applied to other harness architectures that support isolable handoff artifacts, potentially improving credit assignment in multi-step agentic systems.

The extrapolation beyond the training horizon suggests that learned playbook adaptation continues to provide benefits with more attempts, reducing the need for expensive re-evaluation of every candidate.

The method demonstrates that a single policy can be trained to serve multiple roles in a fixed scaffold, which may simplify deployment of self-improving agents.

If ACE-RL scales to larger models and more open-ended tasks, it could enable agents that iteratively refine their own strategies without manual prompt engineering or external memory management—a step toward autonomous recursive self-improvement.

The verifiable-playbook assumption is brittle: if the task verifier is noisy, stochastic, or expensive, the single-rollout estimate of a candidate's value may miscredit the handoff, and the paper's per-role groups cannot fix a biased reward signal.

The edits-discarded control shows the playbook itself drives performance gains beyond actor improvement; this implies that the gap between actor-only and experience-handoff training may widen as tasks require more structured knowledge (API dependencies, implied constraints) that a playbook can capture.

The periodic reset of playbooks (every F=3 encounters) bounds the age of retained experience; extending to cross-task or cross-episode transfer (longer-lived playbooks) could compound the benefit but also risk accumulating stale or misleading tips.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. As a concrete testbed, we fix the Agentic Context Engineering (\ace) harness and train one model to act, reflect, and curate task-local playbooks. Our method, \acerl, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. Across agentic benchmarks, \acerl achieves higher held-out performance than baseline methods. The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-11-13

The authors' main claim, as the classifier picked it out 1.00

“The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit.”

How the main claim is stated: hedged possibility 1.00

What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.32 · human participants 0.03 · controlled comparison 0.91 · randomized assignment 0.02 · states quantitative result 0.06 · compares with prior results 0.95 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
It is common to train Large Language Models (LLMs) for agentic tasks with Reinforcement Learning (RL) from scalar rewards. background 1.00 0.05 0.02 asserted 1.00
However, this throws away the fine-grained information about the environment that the agent experienced during the trial (e.g., error logs). background 0.53 0.10 0.02 asserted 1.00
To utilize this rich information, recent works wrap the agent in a harness that refines its own inputs through reflection or compaction and hands past experience to future rollouts, but the construction of the handed-off artifacts (e.g., a playbook) is itself not learned. gap 0.66 0.04 0.04 asserted 1.00
We want the best of both worlds: gradient updates from a scalar reward, while also utilizing environment information to hand off experience. gap 0.55 0.10 0.07 asserted 0.63
As a concrete testbed, we fix the Agentic Context Engineering (\ace{}) harness and train one model to act, reflect, and curate task-local playbooks. method 0.93 0.91 0.03 asserted 0.92
Our method, \acerl{}, evaluates candidate experience-handoff artifacts by their downstream effect on future actor rollouts, and co-optimizes the role-specific behaviors (action generator, insight reflector, and playbook curator) under a fixed scaffold. method 0.95 0.94 0.10 asserted 0.98
Across agentic benchmarks, \acerl{} achieves higher held-out performance than baseline methods. result 1.00 0.95 0.16 asserted 1.00
The results suggest that agent training should optimize not only the actions in the current rollout, but also the model-written experience that future rollouts inherit. interpretation 1.00 0.79 0.90 hedged possibility 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.92

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-13: 1 request, 6,337 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (11)
Figure 1 of the paper
Figure 1. Experience-Handoff RL. (a) Outcome-only RL compresses each trial into a scalar reward. (b) A harness runs several calls to the same model (cis; in ACE, actor, reflector, and curator) and preserves experience as a model-written context, such as a playbook, handed to the next attempt (blue), but the model never learns to write better playbooks. (c) Experience-handoff RL trains the model for the harness it runs in, from task outcomes alone. For example, ACE-RL rewards each handoff artifact for what it does to a future rollout: (i) each new context is scored by a fresh rollout, and (ii)…
Figure 2 of the paper
Figure 2. Rollout of Qwen3-4B-Instruct-2507 (or checkpoints trained from it) on AppWorld, with
Figure 3 of the paper
Figure 3. Sampling tree of ACE-RL with group sizes N + M + C − 1, M, C for actor, reflector, and curator. (a) Shaded boxes are comparison groups: M reflections of a uniformly chosen seed τj ⋆ and C edits of a uniformly chosen reflection zh share an input; every other reflection receives one edit. Each edit is applied to a copy of P and executed once, and these rollouts join the N seed rollouts of (q, P) in the actor group (purple). (b) Candidate values reward the edit that produced them and, averaged, its reflection; advantages are normalized within each group and all roles share one update.…
Figure 4 of the paper
Figure 4. Held-out test reward against training compute (cumulative update FLOPs, including the critic for PPO-based methods): pass@3 for PPO, GRPO, and ERL; RSI@3 for delayed-joint PPO and ACE-RL. Curves end at the compute budget (Appendix F.1). Each curve starts at the base model at 0 EFLOPs, evaluated the same way, except GRPO on MiniSWEBench; there ACE-RL’s steps 5, 15, 25, and 35 are from a second seed (Appendix F.1). Bands: 1 SEM over tasks. next attempt’s reward Rq(˜τi), turned into GAE advantages by a critic initialized from the policy, without group normalization or reward-based…
Figure 5 of the paper
Figure 5. Actor-only GRPO run in the harness: test RSI@3 against training compute, with ACE-RL’s role prompts. Bands: 1 SEM over tasks. sequential visits in the harness for the methods trained through it (RSI@3). ACE-RL has the highest test reward on all four benchmarks. Actor-only RL and ERL. Paired over tasks, ACE-RL’s RSI@3 is above GRPO’s pass@3 by 0.089±0.015 on Mastermind, 0.120±0.026 on MBPP, 0.042±0.041 on AppWorld, and 0.095±0.034 on MiniSWEBench (±1 SEM). ERL, which hands its revised prompts off only through the weights, is below GRPO on all four benchmarks. PPO is also below GRPO…
Figure 6 of the paper
Figure 6. Best reward after K attempts at the checkpoints of
Figure 7 of the paper
Figure 7. Edits-discarded control. ACE-RL’s RSI@K with edits applied minus discarded (each visit then reads the empty playbook), paired over tasks; above zero, the playbook helps. At K = 1 both read the empty playbook, so the difference is drawn at zero; dotted: training horizon F = 3. Checkpoints as in
Figure 8 of the paper
Figure 8. Delayed-joint ACE-RL versus ACE-RL: held-out test reward (RSI@3) against training EFLOPs, up to the compute budget. Both curves start at the base model (0 EFLOPs). Bands: 1 SEM over tasks
Figure 9 of the paper
Figure 9. RSI@K of both designs at the checkpoints of
Figure 10 of the paper
Figure 10. Training reward: mean reward of the rows in each trainer’s update batch (centered mean over five steps). Actor-only methods act with an empty playbook. For ACE-RL and delayed-joint PPO the batch also contains reflector and curator outputs, scored by candidate values; ERL’s batch contains its guided retries. Values are therefore not comparable across methods. The x-axis is training EFLOPs, up to the compute budget; stopped runs end at their last logged step. 44
Figure 11 of the paper
Figure 11. Critic diagnostics of PPO and delayed-joint PPO from the training logs (ten-step centered mean), against training EFLOPs up to the compute budget. Top: explained variance of the value function on the training batch, clipped below at −1. Bottom: value loss (log scale). Delayed-joint PPO’s explained variance is above 0.95 late in each run, after its reward has collapsed (Section 5.1), so a well-fit critic does not prevent the collapse. 45

Figures are the authors', reproduced from the paper.

References (92)
  1. [1] The secret code is a sequence of digits. Each turn tells you the code length and the valid digits (0 to num_digits-1) on Pith
  2. [2] The same digit may appear more than once in the secret code on Pith
  3. [3] URLhttps://arxiv.org/abs/2506.13131. arXiv preprint arXiv:2506.13131. Qwen Team. Qwen3 technical report, 2025. arXiv preprint arXiv:2505.09388. Jürgen Schmidhuber. Gödel machines: Self-referential universal problem solvers making provably optimal self-improvements, 2003. arXiv... on PithPith's reviewarXiv
  4. [4] URLhttps://arxiv.org/abs/2508.19828. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. In Advances in Neural Information Processing Systems,... on PithPith's reviewarXiv
  5. [5] A guess with the wrong number of digits, an invalid digit, or no guess line is rejected with an error. A rejected guess still uses up a turn and earns nothing on Pith
  6. [6] arXiv preprint arXiv:2308.10144. Yujie Zhao, Lanxiang Hu, Yang Wang, Minmin Hou, Hao Zhang, Ke Ding, and Jishen Zhao. Stronger-MAS: Multi-agent reinforcement learning for collaborative LLMs, 2025. URL https: //arxiv.org/abs/2510.11062. Yifei Zhou, Andrea Zanette, Jiayi Pan, Se... on PithPith's reviewarXiv
  7. [8] Do any of your operations target the same tip or the same reasoning step? If yes, merge them into one operation. ## Current Playbook [PLAYBOOK BEGINS HERE] {{current_playbook}} [PLAYBOOK ENDS HERE] ## Reflector’s Latest Insights {{guidebook}} ## Output Format Complete the chec... on Pith
  8. [9] After each guess, you receive feedback: - exact_matches: number of digits with the correct value AND the correct position - value_matches: number of digits with the correct value but the WRONG position on Pith
  9. [10] Rewards: - Solving the complete code earns reward 1. - Otherwise each exact match earns 1/code_length and each correct value in a wrong position earns 0.5/ code_length. - Your score for the game is your BEST single guess. A new guess can never lower your score, so every turn s... on Pith
  10. [12] Read **Current Playbook** to help you solve the game. ## Current Playbook You are provided with a playbook containing strategies learned from earlier attempts at this same game. [PLAYBOOK BEGINS HERE] {{ playbook }} [PLAYBOOK ENDS HERE] ## Strategy Reminders - Start with a gue... on Pith
  11. [13] **Trajectory Summary**: The actual result, whether the Generator solved the code, how many turns it used, and the best reward it reached on Pith
  12. [14] **Trajectory Record**: A guess table followed by the Generator’s replies. - The guess table is computed from the environment and is always correct. Each row shows the guess as the environment read it, its exact and value feedback, and its reward. A row marked REJECTED scored n... on Pith
  13. [16] How many tips are in the Current Playbook? Aim for at most 5 tips. When the book is longer, merge exact duplicates while retaining their useful facts. Do not delete a correct tip merely to meet this target; future attempts still need the same evidence. ### Step 2: Evaluate eac... on Pith
  14. [18] If your operation is ADD, does it include only ‘type‘ and ‘content‘ with NO ‘bullet_id‘? on Pith
  15. [19] If your operation is UPDATE, does it include ‘type‘, ‘bullet_id‘, and ‘content‘? on Pith
  16. [20] If your operation is REMOVE, does it include only ‘type‘ and ‘bullet_id‘? on Pith
  17. [21] Is your content a plain JSON string containing the tip itself, free of tip IDs like ‘[1]‘, labels such as "Tip :" or "Tip content:", and insight/case classifications such as "Insight 1: Case 3"? The system assigns IDs automatically. Classifications and operation explanations b... on Pith
  18. [22] Does any tip you ADD or UPDATE name the secret code, a digit the code contains, or the position of a digit? If yes, rewrite it as a rule for reasoning, or drop the operation on Pith
  19. [25] Read the problem description carefully on Pith
  20. [26] Write clean, correct, simple Python code that solves the problem on Pith
show 72 more
  1. [27] Your code will be tested against test cases on Pith
  2. [28] If tests fail, analyze the error messages and revise your code on Pith
  3. [29] Read **Current Playbook** to help you answer the question. ## Current Playbook You are provided with a playbook containing strategies learned from solving the same problem. [PLAYBOOK BEGINS HERE] {{ playbook }} [PLAYBOOK ENDS HERE] ## Important Programming Reminders - Make sur... on Pith
  4. [30] **Trajectory Summary**: The actual result, whether the Generator succeeded or failed, how many turns it used, and why it stopped. The test cases are NEVER wrong on Pith
  5. [31] **Trajectory Record**: The full record of the Generator’s attempt. This contains two kinds of information: - The Generator’s own reasoning and code (which may contain wrong assumptions) - The test results and error messages from the environment (which are actual outcomes) You ... on Pith
  6. [32] Can you identify tips in the **Current Playbook** section? If the playbook is blank, note that it is empty on Pith
  7. [33] How many tips are in the Current Playbook? Aim for at most 10 tips. When the book is longer, merge exact duplicates while retaining their useful facts. Do not delete a correct tip merely to meet this target; future attempts still need the same evidence. ### Step 2: Evaluate ea... on Pith
  8. [34] If the playbook is empty, did you choose UPDATE or REMOVE? That’s invalid -- the only valid operations on an empty playbook are ADD or no operation on Pith
  9. [35] If your operation is ADD, does it include only ‘type‘ and ‘content‘ with NO ‘tip_id‘? on Pith
  10. [36] If your operation is UPDATE, does it include ‘type‘, ‘tip_id‘, and ‘content‘? on Pith
  11. [37] If your operation is REMOVE, does it include only ‘type‘ and ‘tip_id‘? on Pith
  12. [38] Is your content free of tip IDs like ‘[1]‘? The system assigns IDs automatically on Pith
  13. [39] Does every REMOVE cite concrete evidence that the tip is wrong or harmful, or identify an exact duplicate retained elsewhere? Similar subject matter alone is insufficient. Preserve complementary facts and correct tips used in a successful attempt. 26 Experience-Handoff Reinfor... on Pith
  14. [40] Do any of your operations target the same tip or the same characteristic? If yes, merge them into one operation. ## Current Playbook [PLAYBOOK BEGINS HERE] {{current_playbook}} [PLAYBOOK ENDS HERE] ## Reflector’s Latest Insights {{guidebook}} ## Output Format Complete Steps 1 ... on Pith
  15. [41] To do this, you will need to interact with apps (e.g., spotify, venmo etc) using their associated APIs on my behalf. For this you will undertake a *multi-step conversation* using a python REPL environment. That is, you will write the python code and the environment will execut... on Pith
  16. [42] Here are three key APIs that you need to know to get more information: - To get a list of apps that are available to you print(apis.api_docs.show_app_descriptions()) - To get the list of apis under any app listed above, e.g. spotify print(apis.api_docs.show_api_descriptions(ap... on Pith
  17. [43] Read **Current Playbook** to help you complete the task on Pith
  18. [44] When you have completed the task, call ‘apis.supervisor.complete_task()‘ to finish: - If the task asks for information (e.g., "how many", "give me a list of"), pass the raw value as the answer: ‘apis.supervisor.complete_task(answer=79)‘ or ‘apis.supervisor.complete_task(answer... on Pith
  19. [45] Write ONE small code block per step. Wait for the output before writing the next step on Pith
  20. [46] Always check API specs using apis.api_docs.show_api_doc() before calling any API. Do not guess API names or parameters on Pith
  21. [47] Use the "supervisor" app for your account info and credentials. Use the "phone" app for contacts on Pith
  22. [48] If an API call fails, use apis.api_docs.show_api_descriptions() to discover what APIs exist. Never repeat failing code on Pith
  23. [49] Many APIs return items in pages. Loop over ‘page_index‘ to get all results on Pith
  24. [50] When calling complete_task(), follow Environment Rule 4: pass the raw value for information tasks (e.g., answer=79, answer="Song A, Song B"), or call with no argument for action tasks. Never wrap answers in sentences on Pith
  25. [51] Before calling complete_task(), always print and verify your result. If the result is empty or seems wrong (e. g., 0 items found), investigate before submitting on Pith
  26. [52] Do NOT call complete_task() until you have successfully fetched and printed the data you need on Pith
  27. [53] Playbook contains task-specific tips. Apply a tip if it clearly matches your current situation on Pith
  28. [54] Playbook tips may be inaccurate or incomplete. Always verify against the actual API documentation before following a tip. ## Task Using these APIs, generate code to solve the actual task. Start by discovering the available APIs. You can refer to playbook when necessary. My nam... on Pith
  29. [55] Read the original task before the playbook or trajectory. Identify its entities, source of information, required actions, selection conditions, and action order. Keep every required part of the result. Then read the summary: outcome, submission status, turns used, and terminat... on Pith
  30. [56] Find the earliest unresolved mismatch between the requested result and the actual attempt. Check whether an existing tip introduced that mismatch. Quote the task requirement and the exact tip, code, documented field, or short environment reply that supports your diagnosis. Dis... on Pith
  31. [57] State what the next attempt must do differently and the condition that must hold before submission. Correct a contradicted tip rather than adding checks around its wrong premise. Give the corrected operation, predicate, or code fragment when it follows from the supplied eviden... on Pith
  32. [58] Identify the current tip IDs and the number of tips. An empty playbook has no valid UPDATE or REMOVE targets. 30 Experience-Handoff Reinforcement Learning for Recursive Self-Improvement on Pith
  33. [59] Compare the task’s entities, information source, required actions, conditions, and action order with the playbook and each insight. Correct a tip that demonstrably contradicts the task instead of strengthening its wrong premise. For each insight, identify the observed action o... on Pith
  34. [60] Compare supported insights with existing tips. Preserve helpful facts and prioritize the correction for an observed failure. Aim for 3-5 concise, nonduplicative actionable tips; keep fewer when fewer lessons are supported. Prefer one concise UPDATE that merges related details ... on Pith
  35. [61] Use ADD for a supported fact that is missing. Use UPDATE for a concrete correction or useful operational detail. Use REMOVE only when the original task or cited evidence shows the tip is incorrect, fully duplicated, or irrelevant. Merge useful details before removing a duplicate on Pith
  36. [62] A success alone never justifies REMOVE. A useful tip followed during a successful attempt should be retained. A failure alone does not show an existing tip is incorrect; the agent may not have followed it on Pith
  37. [63] Keep the exact supported operation, data scope, predicate, or initialization/reset rule in the tip instead of weakening it to "use the right API" or "verify the answer." Include the concrete condition to check before submission when the evidence supplies one. Use only module n... on Pith
  38. [64] Do not add a prerequisite, account rule, or stopping rule without support from the original task or cited API evidence. A failed route does not prove the goal is impossible, and a failure explanation does not replace the requested action. Failed attempted values are not verifi... on Pith
  39. [65] If the insight is empty or generic, do not add or rewrite tips from speculation. You may correct or remove an existing tip when its text demonstrably conflicts with the original task; state the exact conflict in reasoning. Otherwise, if no supported new fact or correction chan... on Pith
  40. [66] The GitHub issue the Generator was asked to resolve on Pith
  41. [67] A Trajectory Summary computed from the environment and the grader. It is always correct. It contains the Outcome, the grader’s Verdict, whether the Generator submitted, the turns used out of the turn limit, tool- call counts, edit outcomes, the turn of the first successful edi... on Pith
  42. [68] The playbook the Generator was given for this attempt on Pith
  43. [69] The Trajectory Record: each turn’s action followed by the environment’s reply. Replies are accurate but cut after about 1500 characters. The Generator’s own reasoning is not shown. An ideal attempt follows a 5-phase workflow within the turn limit: Locate - Reproduce - Apply fi... on Pith
  44. [70] What is the GitHub issue asking for, in one sentence? on Pith
  45. [71] Did the Generator submit? How many turns were used? on Pith
  46. [72] Turn of the first successful edit, and the Repeated Actions line on Pith
  47. [73] Patch Produced, and the FAIL_TO_PASS / PASS_TO_PASS counts if present. Step 2 - Trace the trajectory and answer the following questions: on Pith
  48. [74] At which turns did the Generator locate, reproduce, apply, verify, and submit? Which phases never happened? on Pith
  49. [75] Where does the bug live? Name the file and function the Generator found (or should have found, judging from the issue and the replies) on Pith
  50. [76] If SUCCESS: what located the code quickly, and what edit made the tests pass? What may not be reliably reproduced next time? on Pith
  51. [77] If FAILED: - At which turn, and in which phase, did the Generator first go wrong? - Did it repeat an action, view a whole large file or grep the whole repository and receive a cut-off reply, or keep exploring without editing? Quote the action. - Did its edits fail (e.g. "old_s... on Pith
  52. [78] Which playbook tip, if any, did the Generator follow? Did it help, or did it send the Generator the wrong way? Give the tip’s number in brackets, e.g. [2]. Step 3 - Determine the Case from the Verdict: - Case 1: Fixed cleanly. Verdict RESOLVED, and the Generator located, fixed... on Pith
  53. [79] Can you identify tips in the <CURRENT PLAYBOOK> section? If the playbook is blank, note that it is empty on Pith
  54. [80] How many tips are in the Current Playbook? Count them carefully -- never guess. Aim for at most 5 tips. When the book is longer, merge exact duplicates while retaining their useful facts. Do not delete a correct tip merely to meet this target; future attempts still need the sa... on Pith
  55. [81] If the playbook is empty, did you choose UPDATE or REMOVE? That is invalid -- on an empty playbook the only valid operations are ADD or no operation on Pith
  56. [82] If an operation is ADD, does it include only ‘type‘ and ‘content‘, with NO ‘bullet_id‘? on Pith
  57. [83] If an operation is UPDATE, does it include ‘type‘, ‘bullet_id‘, and ‘content‘? on Pith
  58. [84] If an operation is REMOVE, does it include only ‘type‘ and ‘bullet_id‘? on Pith
  59. [85] Is your content a plain JSON string containing the tip itself, free of tip IDs like ‘[1]‘, the "Verdict: ..." prefix, labels such as "Tip:", and insight/case classifications such as "Insight 1: Case 3"? The system assigns IDs automatically. Classifications and operation explan... on Pith
  60. [86] Does every REMOVE cite concrete evidence that the tip is wrong or harmful, or identify an exact duplicate retained elsewhere? Similar subject matter alone is insufficient. Preserve complementary facts and correct tips used in a successful attempt on Pith
  61. [87] Do any of your operations target the same tip or the same characteristic? If yes, merge them into one operation on Pith
  62. [88] Does any tip you ADD or UPDATE name an exact test or contain exact code? If yes, fix it -- tip content must never leak the gold tests or the solution on Pith
  63. [89] Is each tip at most two sentences, and does it name where to look (file or function) or what behavior to change, or a concrete first action? If not, make it concrete or drop it on Pith
  64. [90] Is your tip written from the insights rather than copied from this prompt’s placeholders or generic rules the Generator already has (narrow grep, view_range, never repeat a call)? Spend the tip’s words on this bug. </TASK> <CURRENT PLAYBOOK> [PLAYBOOK BEGINS HERE] {{current_pl... on Pith
  65. [91] The code must identify and like all transactions to and from roommates for the ongoing year, not just one on Pith
  66. [92] Check before submission: The total number of transactions to like must be greater than zero and all transactions identified must be liked before submission on Pith
  67. [93] The code must process all transactions to and from roommates by checking both direc- tions (sent to and received from) for each roommate and ensure all are included in the like operation on Pith
  68. [94] Check before submission: The response from like_transaction for each transaction must be successful, and the total count of liked transactions must match the total count of transactions to and from roommates. Visits 2–4 complete the task. Mastermind (step 300; task mastermind_... on Pith
  69. [95] Insight 1 suggests the next best guess could be a 1 in the second position to ensure all constraints are met. This provides a clear direction for the Generator to narrow potential code candidates. Visits 2–4 solve the game. H LIMITATIONS ACE-RL requires a harness whose handoff... on Pith
  70. [2024] arXiv preprint arXiv:2407.21787. Yuzheng Cai, Siqi Cai, Yuchen Shi, Zihan Xu, Lichao Chen, Yulei Qin, Xiaoyu Tan, Gang Li, Zongyi Li, Haojia Lin, Yong Mao, Ke Li, and Xing Sun. Training-free group relative policy optimization, on PithPith's reviewarXiv
  71. [2025] URLhttps://arxiv.org/abs/2510.08191. arXiv preprint arXiv:2510.08191. Aaron Chan, Ahmed Shalaby, Alexander Wettig, et al. Composer 2 technical report, 2026. URL https://arxiv.org/abs/2603.24477. arXiv preprint arXiv:2603.24477. Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An.... on PitharXiv
  72. [2026] URLhttps://arxiv.org/abs/2601.16175. arXiv preprint arXiv:2601.16175. Eric Zelikman, Eliana Lorch, Lester Mackey, and Adam Tauman Kalai. Self-taught optimizer (STOP): Recursively self-improving code generation, 2023. arXiv preprint arXiv:2310.02304. Jenny Zhang, Shengran Hu, C... on PithPith's reviewarXiv

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment