{"id":"7baf7e58-ca28-4b42-b235-e5acaf7163d4","arxiv_id":"2607.14777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"SEED improves agentic RL by converting a policy's own completed trajectories into natural-language skills and distilling the skill-induced probability shift back into the policy as dense token-level supervision.","lead":"SEED trains LLM agents by having the policy itself turn finished interaction traces into natural-language 'hindsight skills' and then distilling those skills' behavioral effect back into the policy during reinforcement learning. It reports large gains over GRPO-style outcome-only RL on ALFWorld, WebShop, search-based QA, and two vision tasks, with the same model serving as both actor and skill analyzer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SEED's benefit hinges on self-generated skills being behaviorally informative (Eq. 8), a condition the paper states but never tests; without a skill-content control, the OPD gains may be regularization or SFT rather than skill-guided credit.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the paper's own theory reduces the claimed benefit to an unproven positive covariance between skill-induced gate weights and action values (Eq. 8). I agree that this is the central condition for the mechanism to be genuinely 'skill-guided credit assignment' rather than a generic regularizer. The empirical section is otherwise substantial—consistent gains across backbones/domains, honest theory, released code—but no experiment directly manipulates skill informativeness. The proposed shuffled-skill control is a decisive, cheap test. Since the reader already issued CONDITIONAL and my concern reinforces rather than changes that verdict, I keep CONDITIONAL (UNCHANGED), with the additional condition that the authors provide a skill-content control before the mechanism claim is accepted at face value.","tokens_in":37979,"tokens_out":11649,"duration_ms":106932,"concrete_test":"On Qwen2.5-3B ALFWorld, rerun Stage 2 with skills randomly permuted across trajectories within each rollout group, keeping all hyperparameters and compute budget identical; compare matched-SEED vs shuffled-SEED vs GRPO-only. If shuffled-SEED stays within ~2 points of matched-SEED while both beat GRPO, the skill-content mechanism is not load-bearing and the central claim should be weakened. If matched-SEED exceeds shuffled-SEED by roughly the 7-point margin seen in the static-skill ablation, the informativeness condition is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"SEED's claimed dense-credit mechanism is only beneficial when Cov_{v~π_k}(Q_k(c,v), w_k(c,v)) > 0 (Eq. 8, Appendix A.1). Proposition 2 proves only that the OPD gradient is nonzero when the gate varies; the paper itself says this is 'informativeness, not automatic correctness' and that the propositions 'do not by themselves imply monotonic return improvement.' Appendix E concedes that self-generated supervision 'can inherit model errors and plateau below oracle-supervised training' and that an inaccurate analysis 'may turn a recurring policy error into an apparently reusable rule.' Because the actor and analyzer share parameters, there is no structural guarantee that the skill-induced gate correlates with actual action value. The Table 2 ablations do not close this gap: removing OPD (87.0 vs 91.8) and replacing on-policy skills with a static library (84.4 vs 91.8) show that pipeline components matter, but a static library is still informative; neither condition removes skill content. Moreover, the 75.0→87.0 gain from SFT initialization alone is larger than the 87.0→91.8 OPD gain, so confounds such as added KL regularization or context diversity could explain part of the auxiliary signal's contribution. A matched-versus-shuffled skill control is needed to attribute the gain to skill informativeness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SEED presents a two-stage self-distillation method for agentic LLM RL. Stage 1 uses externally annotated offline trajectories to SFT a policy into a hindsight-skill generator; Stage 2 repeatedly collects on-policy rollouts with the frozen checkpoint, analyzes them into natural-language skills with the same checkpoint, and jointly optimizes GRPO with a gate-weighted token-level distillation loss that re-scores sampled actions under ordinary and skill-augmented contexts (Section 3.3, Eq. 1). The paper reports consistent gains over GRPO and several self-distillation baselines on ALFWorld, WebShop, Search-based QA, and, in the appendix, two vision-based benchmarks, across three Qwen backbones. The theoretical appendix proves structural properties (occupancy-matched target, dense signal under tied rewards, analyzer-staleness bound) and explicitly disclaims monotonic improvement without skill informativeness.","tokens_in":38268,"tokens_out":9141,"duration_ms":77959,"significance":"If the empirical claims hold, SEED would be a practical and inexpensive way to provide denser credit in sparse-reward agentic RL using only self-generated natural-language skills, with no additional inference-time cost. Strengths of the submission include a clear algorithm description, release of code, a detailed implementation appendix, an honest theoretical analysis that states its limitations, and multi-backbone/multi-domain experiments. The main open issues are (i) arithmetic inconsistencies in the headline aggregate results and (ii) the absence of any experiment that isolates skill content from other effects of the auxiliary loss; both must be resolved before the central claim is supported.","major_comments":[{"comment":"The ALFWorld 'Avg' column is not the unweighted mean of the six per-category success rates defined in Eq. (16). For Qwen2.5-3B, the GRPO row has per-category values whose mean is 70.7, not the reported 75.0; OPSD computes to 23.3, not 28.1; and Skill-GRPO computes to 59.9, not 60.2. Because the headline gains in §4.2 ('14.9-45.9 points over GRPO') are measured on these aggregates, all reported averages need to be recomputed and corrected or the main results table cannot be trusted.","section":"Table 1, Eq. (16)"},{"comment":"The central mechanism is not directly tested. Eq. (8) shows that the skill-reweighted target is better than the current policy only when Cov_{v~π_k}(Q_k(c,v), w_k(c,v)) > 0. Proposition 2 (Eq. 11) proves only that the OPD gradient is nonzero when the gate is non-constant; the paper itself calls this 'informativeness, not automatic correctness,' and Appendix E concedes that self-generated skills 'can inherit model errors.' The Table 2 ablations always retain informative skill content, and the w/o-OPD condition (SFT+GRPO, 87.0) already provides a 12-point gain over GRPO (75.0), which is larger than the additional 4.8 points from OPD. A matched control with shuffled, random, or otherwise corrupted skills is necessary to attribute the gains to skill informativeness rather than to added KL regularization or context diversity.","section":"Appendix A.1, Eq. (8); Table 2"},{"comment":"The OPD target is self-referential by construction: the same network acts as rollout actor, trajectory analyzer, and teacher, and Proposition 1 shows that the auxiliary gradient is a KL toward a skill-reweighted version of the model's own policy. Since skill quality is only format-checked (§3.2), and since the teacher log-probabilities are detached but generated from the same model, the auxiliary signal can reinforce existing biases. The paper should report evidence that the value-alignment covariance in Eq. (8) is actually positive during training, or provide an ablation with random/shuffled skills, to establish that the dense signal tracks action quality rather than self-agreement.","section":"Section 3.3, Eq. (1); Appendix A.1, Eqs. (4)-(5)"}],"minor_comments":[{"comment":"The proof refers to 'the clipped reward-driven term in Eq. 3.3,' but no equation number 3.3 exists; it should reference the displayed GRPO loss in Section 3.3.","section":"Proposition 2, Appendix A.2"},{"comment":"Results are reported as point estimates without standard deviations, confidence intervals, or number of seeds. This is especially relevant for Search-based QA, where some claimed gains are as small as 1.4 points; please add run-level variance or at least clarify whether one seed was used.","section":"All main tables"},{"comment":"The abstract and conclusion emphasize vision-based agentic tasks, but the main experiments are text-only; the vision results appear only in an appendix. Consider moving a compact version of Table 8 into the main text or tempering the claim.","section":"Abstract and Appendix C.3"},{"comment":"The static-library condition is not fully specified: the number, format, and source of the static skills should be described to ensure it is a matched comparison to the on-policy skill condition.","section":"Table 2, 'w/o On-Policy Skill'"},{"comment":"The manuscript needs proofreading: the title/header has missing spaces ('SELF-EVOLVINGON-POLICYDISTILLATION FORAGENTICREINFORCEMENTLEARNING'), the author block contains a corrupted symbol ('gl⌢be'), and some arXiv preprint references use future IDs without clearly indicating availability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper has a promising core idea and unusually honest theory, but the arithmetic errors in Table 1 and the missing skill-content control prevent acceptance in the current form. I would request a corrected results table and a shuffled/random-skill ablation; if those come back clean, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague —\n\nThe one thing to know: SEED's synchronized self-evolving loop — the same policy acts as both agent and analyzer, regenerating hindsight skills each iteration and distilling them through a gated on-policy loss — is a real and cleanly implemented mechanism, different from static skill sources or separately maintained teachers. The reported gains are large and consistent across three backbones and five benchmarks. But the empirical support has fixable holes: Table 1 contains internal arithmetic inconsistencies (ALFWorld averages not matching per-category values for several baselines), there are no error bars or multi-seed results, and the OPSD baseline collapses to near zero on Search-based QA, suggesting an implementation artifact.\n\nWhat's genuinely good: the authors are unusually honest about what the theory does and does not show. Proposition 1 states the OPD target is a reweighted version of the current policy; Proposition 2 proves informativeness, not correctness; the appendix explicitly says the propositions do not imply monotonic return improvement. Appendix E concedes that self-generated supervision can amplify recurring errors. The ablations show each component matters. Code and hyperparameters are provided.\n\nThe central weakness is that the load-bearing condition — Eq. 8, Cov(Q, w) > 0 — is stated but never tested. The ablations remove the OPD term or replace on-policy skills with a static library, but neither controls for skill content. A matched/shuffled-skill condition would close the gap. And the SFT initialization alone accounts for a 12-point gain (75 to 87), so part of the benefit could be plain SFT or KL regularization. The authors flag this themselves, so it's a missing control rather than a hidden flaw.\n\nStage 1 uses a proprietary GLM-5.2 annotator, so the full pipeline is not reproducible from public artifacts, though the rest of the implementation is detailed.\n\nWho this is for: anyone working on agentic RL training, credit assignment, or self-distillation. It deserves a serious referee and a conditional accept — the revision should fix the table arithmetic, add seeded variance, audit the OPSD implementation, and run a skill-content control. I'd cite it as related work in my own writing, but I'd be careful not to lean on its quantitative claims until that audit is done.","headline":"SEED's synchronized self-evolving distillation is a genuinely new and sensibly implemented mechanism, but the empirical claims need a table audit, seeded variance, and a skill-content control before I'd take the headline numbers at face value.","tokens_in":38877,"tokens_out":3735,"would_cite":true,"duration_ms":34705,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SEED, a training loop that distills a model's own written hindsight skills into token-level reinforcement-learning signals, claims consistent gains over outcome-only RL across text and vision agentic benchmarks.","keywords":["self-evolving distillation","on-policy distillation","hindsight skills","agentic reinforcement learning","language agents","token-level credit assignment","sparse rewards","sample efficiency"],"falsifier":"Decouple the analyzer from the actor by freezing the trajectory analyzer at an early checkpoint and continuing to train only the policy with the joint objective. If performance does not degrade as the policy evolves, the self-evolving component is not the source of the gains. More directly, replace self-generated skills with random or adversarially wrong skills of the same format; if SEED still shows large improvements, the distillation signal is not carrying the claimed behavioral information.","tokens_in":37732,"feed_emoji":"🤖","tokens_out":4127,"duration_ms":37832,"temperature":0.7,"pith_summary":"This paper proposes SEED, a training loop in which an agent learns from the hindsight skills it writes about its own completed trajectories. Outcome-based reinforcement learning only rewards whole episodes, so the authors add a dense token-level signal: each sampled action is re-scored with and without a self-generated skill, and the skill-induced probability shift is distilled back into the ordinary policy. Because the same model both acts and analyzes, the supervision evolves with the policy instead of going stale. The central claim is that this joint objective improves success rates, sample efficiency, and cross-domain generalization on text and vision agentic tasks.","feed_headline":"SEED outperforms outcome-only RL on every tested agentic benchmark","feed_subtitle":"Self-generated hindsight skills give token-level credit, lifting success up to 39 points and cutting data needs.","key_machinery":"The mechanism is the gated on-policy distillation (OPD) objective: for each on-policy sampled action token, the model re-scores the same token under the original context and under a context augmented with a self-generated hindsight skill; the detached log-probability shift Δ = sg[ℓ_skill − ℓ_θ] is passed through a sigmoid gate g = σ(β_opd Δ), and the loss E[m·g·(sg[ℓ_skill] − ℓ_θ)] pushes the ordinary policy toward skill-endorsed tokens. The self-evolving loop arises because the same model parameterizes both the actor and the trajectory analyzer, so each policy update refreshes the supervisor.","core_discovery":"The paper's central claim is that on-policy hindsight skills—natural-language rules extracted by the policy from its own completed trajectories—can serve as a dense, self-evolving credit-assignment signal when distilled through a confidence gate. Formally, SEED optimizes L_RL + λ_opd E[m·g·(sg[ℓ_skill] − ℓ_θ)], where ℓ_skill is the log-probability of the sampled action under a skill-augmented context, ℓ_θ is the ordinary log-probability, and g = σ(β_opd Δ) gates tokens by the detached log-probability shift. The gate converts variance in skill support into signed token-level credit, so tokens favored by the hindsight skill are promoted and unsupported tokens suppressed. The paper reports cons","pith_inferences":["Because the gate is computed from the same model's own probability shift, the method will tend to reinforce behaviors the model already finds plausible; a natural test is whether SEED's ceiling is set by the analyzer's skill quality, so comparing against a fixed oracle-supervised teacher could reveal the gap.","The covariance condition in Equation 8 suggests a diagnostic: track the correlation between gate weights and per-token value estimates during training; if it turns negative, the distillation signal is actively harmful and the gate should be downweighted.","The staleness bound in Proposition 3 predicts that sensitivity to analyzer lag grows with the gate sharpness β_opd, implying a testable extension: anneal β_opd over training or study its interaction with learning rate.","Since skills are parametric and prompt-free, SEED could complement—not replace—external memory systems: a policy that also has retrieval access might distill retrieved skills into parameters, then drop the retrieval at inference."],"forward_implications":["If correct, hindsight supervision from a model's own trajectories can fill the gap between sparse trajectory rewards and token-level credit assignment, improving success rates by 5.5–39.0 points over outcome-only RL across backbones.","Skills are internalized into parameters, so inference requires no skill prompts, external memory, or retrieval modules; deployment cost is unchanged.","Sample efficiency improves: with only 60% of training data, SEED matches or exceeds full-data outcome-only RL on ALFWorld.","Cross-domain generalization improves: on the ALFWorld unseen split, SEED gains +15.3 points macro-average over the outcome-only baseline, improving in five of six task families.","The self-evolving loop transfers to vision-based agentic tasks, with gains on Sokoban and EZPoints over outcome-only RL."],"fun_headline_variants":["SEED self-distills hindsight skills to beat RL baselines","Self-evolving skills distill into dense token-level RL credit","On-policy hindsight skills beat sparse rewards in agent RL","SEED's self-generated skills give token-level RL credit"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim depends on self-generated hindsight skills being behaviorally informative—specifically, that skill-induced gates assign systematically more support to actions with higher expected return (positive covariance between gate and action value); the paper's propositions establish only that the signal is non-zero and varies across tokens, not that it is correct.","fun_headline_variants_meta":{"raw":{"variants":["SEED self-distills hindsight skills to beat RL baselines","Self-evolving skills distill into dense token-level RL credit","On-policy hindsight skills beat sparse rewards in agent RL","SEED's self-generated skills give token-level RL credit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001115,"raw_usage":{"total_tokens":4514,"prompt_tokens":812,"completion_tokens":3702,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":556,"completion_tokens_details":{"reasoning_tokens":3635}},"tokens_in":556,"tokens_out":3702,"duration_ms":23360,"temperature":1.0,"reasoning_tokens":3635,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T01:04:23.356101+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Decouple the analyzer from the actor by freezing the trajectory analyzer at an early checkpoint and continuing to train only the policy with the joint objective. If performance does not degrade as the policy evolves, the self-evolving component is not the source of the gains. More directly, replace self-generated skills with random or adversarially wrong skills of the same format; if SEED still shows large improvements, the distillation signal is not carrying the claimed behavioral information.","supporting_citations":[],"review_version":1}