{"id":"45ba028e-0644-4e86-b985-ecf409f1b29b","arxiv_id":"2608.09555","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A bidirectional self-distillation signal, built from an augmented Meta-Skill view and a pruned skill view, rescales GRPO advantages and improves skill utilization for LLM agents on ALFWorld and WebShop.","lead":"BCSD is a training method that makes LLM agents follow external text skills more carefully by scoring every action twice, once with extra high-level guidance and once with trimmed instructions, and using the difference between the two scores to adjust reinforcement learning updates. It reports the best overall results among skill-based RL baselines on ALFWorld and WebShop with three Qwen models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central direction-preservation claim assumes A_BCSD is a constant multiplier, but the paper never detaches the token-level gaps; without stop-gradient, the objective's gradient can change per-token update directions and reward-hack the context gap.","rationale":"The reader's concern about ~50% agreement is legitimate: it is consistent with noisy gaps, and the absence of variance reporting makes the empirical advantage less secure. However, the more fundamental issue is the objective itself. The method's selling point is that rescaling preserves the reward-determined update direction. That property is only true if A_BCSD is held fixed during the gradient step. In the paper, Δ_{i,t} (Eqs. 9–11) is computed from log π_θ under C1/C2, and π_θ is the same policy being optimized. Eq. 15 places A_BCSD inside the loss with no indication of detach. If the code backpropagates through Δ, the gradient contains ∂A_BCSD/∂θ. For a positive A_GRPO, this term can be positive even if the token is not actually skill-following; it only needs the policy to become slightly more confident under the augmented context. Conversely, for negative A_GRPO, the term can reduce punishment by making Δ positive. This is a form of objective hacking: the network can change its own advantage weights at no reward cost. The paper even emphasizes that 'BCSD changes only the magnitude of the advantage,' so the entire claim rests on this exogeneity assumption. Without a detached computation, the method is not 'RL with a rescaled advantage' but a different, non-standard objective whose behavior on long-horizon agents is uncontrolled. This is directly testable by code inspection or by the detach/no-detach ablation. I therefore agree with the conditional verdict but locate the decisive condition in the computational graph, not only in the noise level of the gaps.","tokens_in":13519,"tokens_out":8867,"duration_ms":85784,"concrete_test":"Run the released BCSD code twice on Qwen2.5-3B for ALFWorld: once with `A_BCSD.detach()` applied before the loss (i.e., treating the rescaling weight as a constant) and once without. Compare training curves and final success rates. If the two runs differ materially (e.g., >2 points final success or divergent loss), the paper must specify the detached form for the central claim to hold; if they are identical, the implementation already detaches and the concern is resolved. Alternatively, inspect the code for `torch.no_grad()` / `.detach()` around the re-scoring of log-probs in Algorithm 1 line 8.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim—'BCSD changes only the magnitude of the advantage while preserving the update direction determined by the trajectory-level reward'—holds only if A_BCSD in Eq. 13 is treated as an exogenous (stop-gradient) multiplier in the loss of Eq. 15. The paper never states that the token-level log-probabilities used to compute Eqs. 9–11 are detached from the computation graph. Since those log-probabilities are functions of the current policy θ and appear both in the importance ratio ρ_{i,t} and in the advantage A_BCSD, the gradient of L_BCSD w.r.t. θ contains extra terms of the form ∂A_BCSD/∂θ. These terms can increase or decrease the effective per-token update independently of the environment reward. For a positive trajectory-level advantage, the extra term can even flip the sign of the gradient for individual tokens by making the policy more likely under C1 than under C0, artificially inflating w_{i,t}. The stated direction-preservation property is therefore not guaranteed for the implemented objective unless A_BCSD is explicitly detached. This is a load-bearing implementation detail: if the released code does not detach, the reported gains may come from optimizing the context-gap itself rather than from better skill utilization, undermining the central claim. The paper should specify the stop-gradient treatment (e.g., with A_BCSD = A_GRPO * (1-λ_n + λ_n * w_{i,t}.detach())) or provide a derivation of the full gradient showing that the extra terms do not affect the update direction.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes BCSD, a method for training skill-based LLM agents in which the standard GRPO advantage is rescaled token-by-token by a combination of two self-distillation signals: an augmented context that adds a policy-dependent Meta-Skill and a reduced context that prunes the general skill. The rescaling factor is a clipped exponentiated convex combination of the token-level log-probability gaps between each context and the base context. The authors claim that this rescaling changes only the magnitude of the advantage, preserving the update direction determined by the trajectory-level reward. Experiments on ALFWorld and WebShop with Qwen2.5-7B, Qwen2.5-3B, and Qwen3-1.7B report improvements over baselines, and ablations examine the two context views, the coefficient schedule, and the Meta-Skill refresh mechanism.","tokens_in":14023,"tokens_out":4127,"duration_ms":41211,"significance":"If the results hold, BCSD addresses a real gap in skill-based agent training: how to make a policy follow explicit external skills without internalizing them into parameters. The method is clearly formulated, the experimental design covers multiple model scales and two standard benchmarks, and the paper includes useful ablations and a hyperparameter sensitivity analysis. The skill-utilization analysis in Figure 6 is a thoughtful attempt to show that gains come from reliance on the task-specific skill rather than from memorization. However, the central claim of direction preservation depends on an unstated stop-gradient treatment, and the empirical evidence is weakened by the absence of variance information and by hyperparameter selection on the evaluation benchmarks. These issues are load-bearing for the paper's conclusions.","major_comments":[{"comment":"The claim that BCSD 'changes only the magnitude of the advantage while preserving the update direction' is not guaranteed by the equations as written. The loss in Eq. (15) contains the advantage A_BCSD_{i,t} defined in Eq. (13), which depends on w_{i,t} and hence on the token gaps in Eqs. (9)-(11). Those gaps are functions of the current policy parameters θ. Unless the author explicitly detach A_BCSD from the computation graph (e.g., using A_BCSD = A_GRPO * (1 - λ_n + λ_n * w.detach())), the gradient of L_BCSD w.r.t. θ includes extra terms of the form ∂A_BCSD/∂θ. These terms can change per-token update directions independently of the trajectory reward and can even amplify tokens for which the policy is more likely under C1 than under C0, effectively optimizing the context gap rather than skill utilization. The manuscript does not state whether the rescaling is detached, and Algorithm 1 does not clarify this. The authors must either specify the stop-gradient treatment or provide a full gradient derivation demonstrating that the extra terms do not alter the optimization direction.","section":"Methods, Eqs. (12)-(15)"},{"comment":"All reported results come from a single run with no indication of the number of seeds, no error bars, and no significance tests. Given the stochasticity of policy gradient training and the fact that the headline improvements are modest (e.g., 3.1 points on ALFWorld with Qwen2.5-7B, and 0.8 points below the best WebShop score), the aggregate 'strongest overall performance' claim cannot be assessed for reliability. The paper should report mean and variance over at least three seeds, and ideally a paired significance test across tasks.","section":"Experiments, Table 1"},{"comment":"The hyperparameters α and λ_0 are tuned on the same benchmarks (ALFWorld and WebShop) where the final results are reported. Table 3 and Figure 3 present performance under different settings on these test tasks, and the chosen values (α=0.9, λ_0=0.1) are selected because they give the best reported numbers. This constitutes tuning on the test set and inflates the apparent advantage of BCSD over baselines that may not have received the same per-benchmark tuning. The authors should either use a held-out validation split for hyperparameter selection or acknowledge the selection bias and provide a sensitivity analysis over a range of values on unseen tasks.","section":"Experiments, Table 3 and Figure 3"},{"comment":"The near-chance token-level sign agreement ratio (fluctuating between 47% and 53% in Figure 5) is interpreted as evidence that the two context views are complementary. However, it is equally consistent with at least one of the two gaps being dominated by noise. Because the whole method rests on the assumption that the token-level gaps carry reliable, task-relevant credit information, the paper should provide direct evidence that the combined gap is informative: for example, the gap could be compared between successful and failed trajectories, correlated with per-token outcomes, or examined for consistency across checkpoints. Without such analysis, the mechanism underlying the reported gains remains speculative.","section":"Further Analysis, Figure 5"}],"minor_comments":[{"comment":"The implementation details omit the prompts used for the ExtractLLM and PruneLLM calls, the evidence buffer sizes, and the exact definition of a validation window; these details are needed for reproducibility and should be included in an appendix.","section":"Experimental Setup"},{"comment":"The schematic in Figure 2 is dense and hard to follow; in particular, the box labeled '1) Context Construction' mixes equations with qualitative labels, making it difficult to map the figure to Eqs. (6)-(8). A clearer separation of the context construction and the rescaling pipeline would improve readability.","section":"Figure 2"},{"comment":"There is a typographical error in the text 'withaparticularly' (in the paragraph on disabling Meta-Skill updates); it should read 'with a particularly'.","section":"Ablation Studies"},{"comment":"The y-axis of Figure 5 ranges only from 0.47 to 0.53, which visually amplifies small fluctuations; the paper should either rescale the axis to the full [0,1] interval or explicitly note the truncated range in the caption.","section":"Figure 5"},{"comment":"In Algorithm 1, line 8 says 'Re-score the realized tokens under C0, C1, C2 using πθ', but in the main text the gaps are defined using the same policy πθ evaluated at the current parameters; it would be clearer to specify whether this scoring is done under θ_old or θ, since the gradient treatment depends on it.","section":"Algorithm 1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important problem and the method is novel in its use of two complementary context views. However, the unspecified stop-gradient treatment in Eqs. (12)-(15) directly affects the validity of the central 'direction-preserving rescaling' claim, and the single-run, hyperparameters-tuned-on-test evaluation makes the empirical support fragile. I would need to see the authors clarify the gradient treatment and provide multi-seed results, and ideally an additional analysis that the token-level gaps are informative rather than noise, before this is acceptable. The manuscript is currently more in the style of a workshop/conference paper; if the journal expects a more complete archival treatment, the authors should also expand the implementation details and provide a full gradient derivation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: BCSD is a real, incremental contribution to the skill-based LLM-agent RL subfield, and the empirical trend is consistently positive, but the paper needs a multi-seed rerun and an explicit statement about stop-gradient before I'd trust the central claim.\n\nThe new bit is sensible: instead of one privileged context teacher, BCSD builds two views — an augmented context with a policy-derived Meta-Skill and a pruned context with reduced general skill — and uses the token-level log-probability gaps to rescale the GRPO advantage. That bidirectionality, plus gap-weighted rescaling, isn't in the cited OPSD papers. The ablations show both views contribute, and the Figure 6 analysis (bigger performance drop when the task skill is removed) is a nice piece of evidence that the method improves skill usage rather than memorizing task knowledge. If the code ships and the results hold across seeds, this is a useful method.\n\nThe soft spots, in proportion. First, the 'direction-preserving' claim in Eq. 13 is not guaranteed as written. Equations 9–11 compute Δ from the current policy's log-probs, and those log-probs appear in the loss through A_BCSD. Unless those gaps are detached from the computation graph (stop-gradient), the gradient of L_BCSD w.r.t. θ has extra ∂A/∂θ terms that can change per-token update directions independently of the trajectory reward. The paper never says A_BCSD is treated as a constant multiplier. This is a load-bearing implementation detail; I'd want either an explicit .detach() in the released code or a derivation showing the extra terms don't flip directions.\n\nSecond, everything rests on single-run numbers. No seeds, no variance, no error bars. The gains are a few points on two benchmarks, so run-to-run noise could be on that scale.\n\nThird, α and λ0 are tuned on the same benchmarks that get reported as success. That's common in this literature, but it weakens the headline numbers a bit.\n\nFourth, Figure 5 shows the two views agree in sign only about half the time. The paper reads this as complementarity, which is defensible, but it also means each view is a weak signal on its own; with α=0.9, the pruned view is mostly a moderator. That's fine, but it makes the robustness question more pressing.\n\nBottom line: this deserves a serious referee. The method is clearly defined, the experiments are honest in their reporting (Figure 5 is refreshingly non-selective), and the claims are falsifiable. I'd ask for multi-seed results and a clear statement about stop-gradient before accepting.","headline":"A real incremental method for skill-based agent RL with a consistently positive trend, but the direction-preservation claim needs an explicit stop-gradient statement and multi-seed results before I trust it.","tokens_in":14448,"tokens_out":2558,"would_cite":false,"duration_ms":23612,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BCSD rescales the GRPO advantage using token-level gaps from a Meta-Skill-augmented view and a pruned skill view, and reports the strongest overall success rates on ALFWorld and WebShop across three model scales.","keywords":["skill-based LLM agents","self-distillation","reinforcement learning","GRPO","external skill utilization","meta-skill","token-level credit assignment","advantage rescaling"],"falsifier":"Train a control variant of BCSD in which, for each trajectory, the token-level combined gaps $\\Delta_{i,t}$ are replaced by randomly permuted gaps sampled from the same set of magnitudes (so only the token-to-token assignment is destroyed), keeping all other settings identical; if this control matches BCSD's success rate, the specific gap content is not what drives the improvement. A more direct check is to substitute an unrelated text for the Meta-Skill in the augmented view: genuine skill-utilization guidance should collapse BCSD's advantage toward the single-view baseline.","tokens_in":13367,"feed_emoji":"🧭","tokens_out":7987,"duration_ms":61655,"temperature":0.7,"pith_summary":"The paper targets a gap in training LLM agents that receive external natural-language skills: agents often ignore or misuse the guidance. It proposes BCSD, a training method that pairs reinforcement learning with self-distillation from two complementary skill-context views, so the agent learns to apply the provided skills rather than memorize task knowledge. The augmented view adds a policy-dependent Meta-Skill summarizing how the skill should be used, while the pruned view condenses general guidance to focus on the task-specific skill. Token-level gaps between these views and the original context rescale the GRPO advantage without flipping its sign, and the strength of that rescaling decays over training. On ALFWorld and WebShop, BCSD reports the strongest overall success rates across three model scales, and removing the task-specific skill at inference hurts its performance more than it hurts a standard GRPO policy, which the paper reads as evidence of genuine skill utilization.","feed_headline":"Two skill views beat one for RL-trained LLM agents","feed_subtitle":"BCSD rescales RL advantages with token-level signals from added meta-skill and pruned skill contexts, beating single-view baselines.","key_machinery":"The machinery is the bidirectional context pair and the rescaling weight built from it. Given base context $C_0=(Q,G,S_c)$, BCSD constructs $C_1=(Q,G,S_c,M_c)$ by adding a Meta-Skill $M_c$ extracted from successful and failed rollouts, and $C_2=(Q,G_p,S_c)$ by pruning the general skill $G$. For each realized token, it computes the log-probability gaps $\\Delta^{\\text{aug}}_{i,t}$ and $\\Delta^{\\text{pru}}_{i,t}$ between these views and $C_0$, combines them as $\\Delta_{i,t} = \\alpha\\,\\Delta^{\\text{aug}}_{i,t} + (1-\\alpha)\\,\\Delta^{\\text{pru}}_{i,t}$, and forms a clipped multiplicative weight $w_{i,t} = \\operatorname{clip}\\big(\\exp(\\operatorname{sign}(A^{\\text{GRPO}}_i)\\Delta_{i,t}),\\, 1-c,\\, 1+c\\big)$. The final token advantage is $A^{\\text{BCSD}}_{i,t} = A^{\\text{GRPO}}_i\\,\\big[(1-\\lambda_n) + \\lambda_n w_{i,t}\\big]$, with $\\lambda_n$ decaying to zero. This lets the token-level context signals modulate the magnitude of the advantage while never reversing its sign.","core_discovery":"The central claim is that evaluating each trajectory from two opposite directions of skill-context editing yields more reliable token-level credit for skill utilization than any single privileged context. From a base context of query, general skill, and task-specific skill, the augmented view appends a Meta-Skill distilled from the policy's own successes and failures, and the pruned view replaces the general skill with a condensed version. The same policy re-scores the realized tokens under both views; the two log-probability gaps are combined with a weight alpha and used to rescale the group-relative GRPO advantage token by token, with the coefficient linearly decayed to zero over training. The paper argues this preserves the update direction fixed by trajectory rewards while injecting dense guidance on whether each token follows the skills. Experiments report consistent gains over skill-conditioned RL and single-view self-distillation baselines on ALFWorld and WebShop at three model scales from 1.7B to 7B parameters.","pith_inferences":["The near-50% sign agreement between the two token-level gaps suggests the rescaling may act partly as a variance-reducing regularizer rather than a pure credit signal; a direct test would be to compare against GRPO with a token-level entropy or KL bonus of matched strength.","The same bidirectional context trick could transfer beyond skills to any externally supplied natural-language constraints, such as safety rules or instruction hierarchies, by comparing full-context and context-reduced views.","The Meta-Skill extraction might be simplifiable: if a static summary from a handful of successful and failed trajectories suffices, the periodic refresh loop could be dropped, lowering the method's cost.","The larger performance drop when the task-specific skill is removed after training is consistent with the policy learning to attend to the most informative context, but whether that counts as semantic 'use' of the skill would require probing attention or activation patterns."],"forward_implications":["Skill-based agents can be trained to use external skills more effectively without internalizing them, keeping skills explicit, editable, and transferable at inference time.","The advantage-rescaling scheme injects dense token-level credit while preserving the sign of the trajectory reward, so it can be layered onto any group-relative RL objective.","Removing either the Meta-Skill view or the pruned context view degrades performance, and a decaying rescaling coefficient is required to avoid over-relying on privileged guidance late in training.","Because the augmented view is refreshed as the policy evolves, the method tracks the agent's changing skill-utilization patterns during reinforcement learning.","On the two long-horizon benchmarks tested, BCSD achieves the best overall success rates across three model scales, with the largest gains on WebShop success rate."],"supporting_citations":[{"why":"Supplies the GRPO objective whose group-relative advantage BCSD rescales.","marker":"Guo et al. 2025"},{"why":"Defines the hierarchical SkillBank setup and the base context with general and task-specific skills that BCSD builds on.","marker":"Xia et al. 2026"},{"why":"The RLSD baseline that first combines on-policy self-distillation with RL using a single privileged context, which BCSD extends to two views.","marker":"Yang et al. 2026a"},{"why":"SDAR, a baseline combining skill-conditioned self-distillation with RL and adaptive gating, which BCSD compares against.","marker":"Lu et al. 2026"},{"why":"Skill-SD, a skill-conditioned self-distillation baseline for multi-turn LLM agents, used as a comparison.","marker":"Wang et al. 2026a"},{"why":"Provides the ALFWorld benchmark used for evaluation.","marker":"Shridhar et al. 2021"},{"why":"Provides the WebShop benchmark used for evaluation.","marker":"Yao et al. 2022"}],"fun_headline_variants":["Bidirectional context self-distillation improves skill use in RL LLM agents","BCSD rescores RL advantages with meta-skill and pruned contexts","Dual-view self-distillation gives LLM agents denser skill credit","Meta-skill and reduced skill contexts refine RL advantage for agents","BCSD: two skill-context views sharpen token-level credit for LLM agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the token-level log-probability gaps between the augmented, pruned, and base contexts carry genuine information about how well each token follows the provided skills; if those gaps are mostly noise, the rescaling would randomize update magnitudes and the reported gains would not be stable.","fun_headline_variants_meta":{"raw":{"variants":["Bidirectional context self-distillation improves skill use in RL LLM agents","BCSD rescores RL advantages with meta-skill and pruned contexts","Dual-view self-distillation gives LLM agents denser skill credit","Meta-skill and reduced skill contexts refine RL advantage for agents","BCSD: two skill-context views sharpen token-level credit for LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000811,"raw_usage":{"total_tokens":3575,"prompt_tokens":982,"completion_tokens":2593,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":2497}},"tokens_in":598,"tokens_out":2593,"duration_ms":19029,"temperature":1.0,"reasoning_tokens":2497,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:09:43.053361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a control variant of BCSD in which, for each trajectory, the token-level combined gaps $\\Delta_{i,t}$ are replaced by randomly permuted gaps sampled from the same set of magnitudes (so only the token-to-token assignment is destroyed), keeping all other settings identical; if this control matches BCSD's success rate, the specific gap content is not what drives the improvement. A more direct check is to substitute an unrelated text for the Meta-Skill in the augmented view: genuine skill-utilization guidance should collapse BCSD's advantage toward the single-view baseline.","supporting_citations":[],"review_version":1}