{"id":"9f1b139d-afd8-4fdd-b3a7-56d85ddd6ef8","arxiv_id":"2601.15141","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Replacing execution-error segments in RL trajectories with the model's own successful self-corrections improves accuracy on math and code benchmarks while cutting training steps by roughly two-thirds.","lead":"This paper proposes a training trick for small language models that use Python tools: when a model makes an execution error during reinforcement learning, the error is erased from the training log and replaced by the model's own later successful correction. The result is cleaner training data that boosts benchmark accuracy by a few points and reaches similar final performance in about one-third the training steps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GRPO is applied to counterfactual purified trajectories not sampled from π_θold, making Eq. (5)'s importance ratio invalid; the reported gains may be supervised copying rather than RL exploration.","rationale":"I considered the lack of error bars and unreleased code, but those are reproducibility issues, not threats to the internal argument. The off-policy estimator is the point where the math and the algorithm diverge: the paper derives GRPO for trajectories sampled from π_θold (Eq. 5) and then optimizes it on counterfactual sequences. This is a formal gap, not a disagreement with consensus. The paper's own Table 4 (SAAR deactivated at eval) is a good internalization check and supports the claim that the model retains behavior after training, but it does not validate the training objective. My proposed SFT control would isolate whether the purification benefit is an RL effect or a data-augmentation/SFT effect. If the latter, the headline 'boosts Agentic RL' is overstated even though the data-level recipe may still be useful. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":18355,"tokens_out":8129,"duration_ms":94185,"concrete_test":"Run the identical CLEANER-4B pipeline (same SAAR-purified data, 250 steps, batch 128, LR 2e-6) but replace the GRPO term in Eq. (5) with a plain supervised maximum-likelihood loss on the same τ_purified trajectories (no group advantages, no importance ratios). Compare AIME24/AIME25 Pass@1 against the reported CLEANER and DAPO-baseline. If SFT-on-purified matches or exceeds CLEANER, the RL objective is not the source of the gains and the off-policy estimator concern is confirmed; if CLEANER clearly beats SFT-on-purified, the RL signal is doing real work despite the biased ratio.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that CLEANER's self-purified trajectories boost agentic RL — rests on treating Eq. (5) as a valid GRPO update for τ_purified. In §4.3 the log-probabilities of the corrected code are recomputed under the purified context, but τ_purified was never sampled from π_θold; it is a counterfactual rewrite created by SAAR after observing an execution error. The importance ratio ρ_i = π_θ(τ_purified)/π_θold(τ_purified) therefore has the wrong denominator: the actual proposal distribution is q(τ_purified) = Σ_{τ: T(τ)=τ_purified} π_θold(τ), where T is the SAAR transformation, and this is not computed. Without an unbiasedness argument, the GRPO objective is a biased surrogate. A concrete symptom: if SAAR makes most trajectories in a group successful, the group mean reward becomes 1, the standard deviation 0, and all advantages A_i in Eq. (4) are 0 — so the update vanishes. The paper does not report the fraction of groups with nonzero advantage, nor any control separating the RL signal from a pure imitation/SFT loss on the same purified trajectories. The ablation in Table 2 shows the combination works, but it does not identify the mechanism.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CLEANER, a data-level intervention for agentic reinforcement learning with Python tool use. When a rollout contains an execution error that is later corrected within the same trajectory, the Similarity-Aware Adaptive Rollback (SAAR) mechanism constructs a 'purified' trajectory by replacing the failed step with the successful correction. The replacement is shallow (only the code action is replaced) when the erroneous and corrected code are semantically similar, and deep (the whole reasoning step is replaced) otherwise. The purified trajectories are then used to update the policy with GRPO, with log-probabilities of the corrected code recomputed under the purified context using RadixAttention. Experiments on Qwen3-4B-Instruct and Qwen2.5-7B-Instruct report gains over a DAPO baseline on AIME24/25, GPQA, and LiveCodeBench, and claim state-of-the-art efficiency, matching DemyAgent-4B with one-third of the training steps.","tokens_in":18699,"tokens_out":5273,"duration_ms":55810,"significance":"If the method works as described, it would be a practically useful way to reduce the cost of agentic RL for small models, and the promised release of code and processed datasets would support reproducibility. The core idea of eliminating error-contaminated context during collection is simple and plausible. However, the paper currently lacks a valid derivation of the RL update on purified trajectories; the importance ratio in Eq. (5) is computed against a distribution that did not generate the data. Without either a correct estimator or a control separating imitation from RL exploration, the central claim that CLEANER 'boosts agentic RL' is not established. The empirical evaluation is also based on single runs and, for the Qwen2.5 experiments, on an underspecified data filter.","major_comments":[{"comment":"The GRPO update on purified trajectories is not a valid policy-gradient update as written. Eq. (5) requires the group trajectories {τ_i} to be sampled from π_θold. SAAR, however, constructs τ_purified by a counterfactual transformation T of a rollout that was sampled from π_θold: the corrected code c'_t is generated from the error-augmented context h~_t, not from the purified context. The actual proposal distribution for τ_purified is q(τ_purified) = Σ_{τ: T(τ)=τ_purified} π_θold(τ), which is not equal to π_θold(τ_purified). Recomputing log-probabilities under the purified context (as described in §4.3) fixes the conditioning for c'_t but does not correct the denominator of ρ_i; moreover, for the deep-replacement case the auxiliary reasoning r'_aux is also generated from the error-augmented context and its log-probability is not recomputed at all. The objective is therefore a biased surr","section":"§4.3 and Eq. (5)"},{"comment":"The empirical evidence relies on single-run comparisons without error bars or multiple seeds. Differences of 1–3% in AIME/GPQA/LiveCodeBench are within the range of typical seed-to-seed variance for small-model RL. In addition, the Qwen2.5-7B experiments filter 'trivially easy or unsolvable' instances, but the filtering criterion is never defined. The filter is applied only to the Qwen2.5 experiments and not to the Qwen3-4B experiments, making the cross-model comparison asymmetric. The 70/30 curriculum mixing ratio is also introduced only for Qwen2.5-7B. I would ask for at least three seeds per condition, a precise description of the data filter (or removal of the filter), and a justification for the asymmetric treatment of the two model scales.","section":"§5.1 and Table 2"},{"comment":"Key hyperparameters — similarity threshold γ=0.5, retry limit K=3, and the learning rate 2e-6 for the 4B model — are selected by ablations on the target evaluation benchmarks. The DAPO baseline shares the same final configuration, so the head-to-head comparison is fair, but the absolute gains over the 'RL w/ Tools' condition in Table 2 may partly reflect benchmark tuning rather than the SAAR mechanism. The paper should either report a separate validation split for hyperparameter selection or provide a sensitivity analysis showing that the main conclusions hold across a range of γ, K, and learning rates. The claim in the abstract that gains are 'average 6% on AIME, 3% on GPQA, 5% on LiveCodeBench' also does not match Table 2, which reports 4% on GPQA for Qwen3-4B.","section":"§5.3, Table 3, and Appendix A"}],"minor_comments":[{"comment":"The baseline is called 'Demystify-4B' in §5.1 and 'DemyAgent-4B' in Table 1 and elsewhere. Please use one consistent name.","section":"Table 1 / §5.1"},{"comment":"The hardware section refers to a 'Qwen2.5-4B' model, but the experiments use Qwen3-4B and Qwen2.5-7B. This appears to be a typo.","section":"Appendix A"},{"comment":"The 'RL w/o Tools' row for Qwen2.5-7B reports LiveCodeBench-v6 of 1.1, while Table 1 lists the same base model with 15.2 on LiveCodeBench. The evaluation settings differ, but this is not explained; please clarify.","section":"Table 2 and Table 1"},{"comment":"The manuscript is marked 'Work in progress' on every page and contains corrupted font artifacts in Section 3 and Figures 4–5. The unicode fragments and missing figure text make parts of the paper difficult to read. Please copyedit and regenerate the figures.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a 'Work in progress' preprint, and the main concern is not cosmetic. The GRPO update on counterfactually purified trajectories is not derived, and the paper does not provide the controls needed to distinguish RL from imitation. I believe this is fixable within the manuscript's scope: the authors could reframe the method as a data-synthesis technique with an appropriate supervised or offline objective, or add a valid importance-correction argument and the recommended control experiments. I would also urge the editor to require the data-filter details and error bars before considering the paper for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nThe one thing to know: this paper proposes a real new trick. When a model makes a tool call, gets an error, then self-corrects in the same rollout, SAAR rewrites the trajectory by deleting the failed attempt and grafting in the corrected code, using a difflib similarity score to decide whether to replace just the code or the whole reasoning segment. That is not in the cited baselines, and it is the closest thing to a new idea in this line of work.\n\nWhat it does well: the ablations are internally consistent. SAAR adds 3–6 points over a same-config DAPO baseline across AIME, GPQA, and LiveCodeBench; the eval-time deactivation test shows most of the gain survives without the scaffolding; and the recovery experiment (turning SAAR on at step 200) shows it can stabilize a run that had already gone noisy. Those are credible internal signals.\n\nThe soft spot is load-bearing. The GRPO update in Eq. (5) is applied to τ_purified, but that trajectory was not sampled from π_θold. The importance ratio's denominator is the probability of the corrected code under the purified context, yet the corrected code was actually generated from the error-augmented context. Recomputing log-probs with RadixAttention doesn't fix the proposal distribution; it just recalculates a different probability. The update is a biased surrogate and no unbiasedness argument is supplied. The stress-test's extreme case is real: if most trajectories in a group become successes after purification, group mean reward goes to 1, group std goes to 0, and all advantages vanish. The paper never reports how often that happens, nor does it include a control separating the RL signal from supervised imitation of the same purified trajectories. So we can't rule out that the gains are mostly supervised copying of correct code, with the RL machinery contributing little.\n\nSmaller issues: no error bars anywhere, single runs throughout; the Qwen2.5 data filter (\"trivially easy or unsolvable\") is unspecified; hyperparameters like γ=0.5 and K=3 are tuned on the target benchmarks; and the \"one-third training steps\" claim compares against DemyAgent-4B trained for 750 steps rather than a same-config DAPO baseline trained for 750 steps. The code is promised but not shipped.\n\nBottom line: the idea deserves engagement and the paper should get a serious referee, but the current manuscript needs major revision. The authors need to either justify the off-policy estimator or drop the GRPO framing, add multi-seed results, report the fraction of groups with nonzero advantage, and release the code. I'd send it to review with that expectation.","headline":"A genuinely useful trajectory-purification idea for agentic RL, but the GRPO estimator on counterfactual paths is not justified and the empirical shell is too thin to trust the numbers as-is.","tokens_in":19172,"tokens_out":3729,"would_cite":false,"duration_ms":41850,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CLEANER claims that rewriting failed tool-use steps out of reinforcement-learning trajectories — replacing them with the model's own successful corrections — turns noisy agentic training into clean signal, yielding average accuracy gains of","keywords":["agentic reinforcement learning","trajectory purification","self-correction","similarity-aware adaptive rollback","GRPO","tool use","code interpreter","LLM reasoning"],"falsifier":"Train a model on exactly the same SAAR-purified trajectories, but with the GRPO surrogate replaced by plain supervised imitation (max-likelihood on the purified code and reasoning). If the pass@1 gains on AIME24/25 and LiveCodeBench largely persist, the paper's credit-assignment mechanism is not what drives the gains; the method would then be an SFT data-cleaning trick. Conversely, if the gains vanish, the RL-specific claim is supported.","tokens_in":18233,"feed_emoji":"🧹","tokens_out":5342,"duration_ms":50686,"temperature":0.7,"pith_summary":"The paper is trying to establish that the main obstacle to reinforcement learning for small tool-using language models is not missing rewards or exploration strategy, but the accumulation of execution-error context inside training trajectories. It argues that failed code and tracebacks contaminate the model's context and receive spurious positive credit in outcome-only RL, so defeating that noise matters more than adding dense rewards or supersampling. The proposed solution, CLEANER, uses a Similarity-Aware Adaptive Rollback (SAAR) mechanism to detect when the model has corrected its own code within a rollout and rewrite the history to show only the successful version. Trained on these self-purified paths, models internalize correct reasoning rather than error-recovery loops. A sympathetic reader would care because the method is data-level and cheap: it replaces expensive oversampling and bypasses reward hacking, and the paper reports consistent gains across math, science, and code benchmarks.","feed_headline":"Self-purified trajectories lift agentic RL accuracy 6%","feed_subtitle":"CLEANER deletes error-contaminated steps during collection, matching a strong baseline in one-third the training time.","key_machinery":"Similarity-Aware Adaptive Rollback (SAAR): an online trajectory-rewriting mechanism triggered by execution errors. It freezes the history at the failure, asks the policy to produce a corrected code, verifies it, then uses `difflib.SequenceMatcher` similarity between original and corrected code (threshold γ=0.5) to decide whether to swap in only the corrected code or the whole corrected reasoning+code turn. Log-probabilities are recomputed under the purified context (using a KV-cache-reuse attention strategy) so the update is grounded in the causal path, and for the 7B model 30% of trajectories are left un-purified to preserve self-correction ability.","core_discovery":"The paper's central claim is that trajectory-level execution noise — not token-level reasoning error — is the binding constraint in agentic RL for 4B–7B models. CLEANER converts a noisy exploration log into a counterfactual clean trajectory: when the model's code fails and the same rollout later contains a self-correction that executes successfully, SAAR rolls the history back to the failure point and grafts in the corrected code (shallow replacement if code similarity is high, deep reasoning replacement if low), then recomputes the policy-gradient log-probabilities under the purified context. Training on these purified paths suppresses erroneous tool calls, improves pass@1 accuracy, and rea","pith_inferences":["Editorial inference: If the credit-assignment story is right, the same retrospective-purification idea should transfer to other tool calls (search, shell, APIs) and to other policy-gradient objectives; the similarity heuristic may even need to be replaced by a learned or model-based judgment of 'same attempt'.","Editorial inference: The paper's GRPO estimator on `τ_purified` is a counterfactual rewrite rather than a sample from the old policy; until an unbiasedness argument or a control experiment is supplied, the reported gains could plausibly come from supervised copying of corrected code instead of a novel RL exploration signal. A clean test: train on the same purified trajectories with plain behavior ","Editorial inference: Appendix B reports that using the discarded erroneous actions as negative samples (via online DPO with token masking) failed and even caused training collapse; that self-reported negative result is consistent with the paper's framing that the benefit comes from removing noise, not from contrastive learning, but it also leaves open whether a better-shaped negative signal could ","Editorial inference: The 70/30 curriculum mix is a stability lever: the paper keeps 30% raw trajectories for the 7B model to preserve self-correction. Varying this ratio across task difficulty or training stage might further improve the trade-off between error avoidance and error recovery."],"forward_implications":["Execution-failure frequency during RL is a bottleneck; suppressing it via purification yields consistent pass@1 improvements across four benchmarks and two model sizes.","CLEANER reaches roughly the same accuracy as a strong agentic baseline in about one-third of the training steps, implying data quality can substitute for compute in this regime.","The benefit is internalized: with SAAR disabled at evaluation, accuracy drops by only ~0.6–2.5 percentage points, while deploying SAAR at inference adds only 8.8% latency.","Starting from a degraded policy (step 200 of a baseline), SAAR restores stability and improves AIME24/25 accuracy by 5.2% and 1.0%, but does not fully match from-scratch training — so the mechanism belongs in the training loop, not just after."],"fun_headline_variants":["Purifying trajectories gives agentic RL a 6% accuracy lift","CLEANER's rollback trick: cleaner logs, faster RL","Agentic RL gets 6% accuracy boost from self-corrected paths","One-third training steps for SOTA? Just purge error steps","Trajectory self-purification: agentic RL's shortcut to efficiency"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the GRPO objective evaluated on the rewritten 'purified' trajectory is a valid policy-gradient estimator; because τ_purified was never actually sampled from the old policy, and the importance-weight re-computation does not correct the sampling distribution, the update may instead amount to behavior-cloning the corrected code. If that estimator is biased, the claimed RL-superiority and efficiency gains are on weaker footing.","fun_headline_variants_meta":{"raw":{"variants":["Purifying trajectories gives agentic RL a 6% accuracy lift","CLEANER's rollback trick: cleaner logs, faster RL","Agentic RL gets 6% accuracy boost from self-corrected paths","One-third training steps for SOTA? Just purge error steps","Trajectory self-purification: agentic RL's shortcut to efficiency"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000453,"raw_usage":{"total_tokens":2150,"prompt_tokens":813,"completion_tokens":1337,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":557,"completion_tokens_details":{"reasoning_tokens":1243}},"tokens_in":557,"tokens_out":1337,"duration_ms":12999,"temperature":1.0,"reasoning_tokens":1243,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T08:56:46.959554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a model on exactly the same SAAR-purified trajectories, but with the GRPO surrogate replaced by plain supervised imitation (max-likelihood on the purified code and reasoning). If the pass@1 gains on AIME24/25 and LiveCodeBench largely persist, the paper's credit-assignment mechanism is not what drives the gains; the method would then be an SFT data-cleaning trick. Conversely, if the gains vanish, the RL-specific claim is supported.","supporting_citations":[],"review_version":1}