{"id":"1921a816-d6f6-468c-878c-6508c59c1571","arxiv_id":"2608.04788","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"OCSD improves agentic RL by contrasting Full and Observation-Ablated replay views to derive an observation residual that calibrates token-level GRPO updates.","lead":"When AI agents are trained with reinforcement learning, the model usually gets only one reward for a whole episode, so it cannot tell which words led to success. OCSD subtracts a control replay that hides future observations from a replay that includes them, isolating the true feedback and using it to scale token-level updates during training.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing train-time control: the observation-swap experiment in App. D is diagnostic only, so the paper never shows that OCSD's training gains require the true future observation rather than the dual-view contrast itself.","rationale":"The reader's verdict is already CONDITIONAL, and this stress-test reinforces that judgment rather than moving it. The paper has genuine strengths: it identifies a real confound in privileged replay scoring, proposes a parameter-free residual, reports consistent main-table gains across three benchmarks and three model scales, and includes a diagnostic observation-swapping analysis. However, the diagnostic swap only demonstrates that the residual's AUROC for discriminating valid from invalid steps drops when observations are swapped; it does not show that the policy-update signal in Eq. 11 depends on the true observation. The full method compounds step selection, dual-view scoring, and sign-preserving modulation, so an ablation that randomizes the observation content is necessary to attribute the empirical gain to observation calibration. Without such a train-time control, the headline claim that the residual is 'observation-calibrated' is not yet established. The reader's weakest assumption concerned the structural match of the ablated view; this concern is closely related but specifically targets the missing train-time attribution. Keeping the CONDITIONAL verdict (UNCHANGED) is appropriate.","tokens_in":21766,"tokens_out":8668,"duration_ms":111106,"concrete_test":"Run the full OCSD training protocol on ALFWorld and WebShop with Qwen3-4B (three seeds, same hyperparameters, ρ=0.2, β=0.5), but in the Full replay view replace the realized immediate future observation o_{k+1} with a donor observation sampled from another trajectory whose action shares the same operator, exactly as in Appendix D, while keeping the future-action schema and second future observation fixed. Compare final overall success and WebShop score against standard OCSD and GRPO. If the random-observation variant is within one standard deviation of standard OCSD and well above GRPO, the central observation-calibration claim fails this test; if it falls back toward GRPO, the claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the residual in Eq. 6 isolates observation-specific support and that using it in Eq. 11 improves agent performance because it aligns with local feedback. The load-bearing gap is that the paper never tests this dependence at training time. Appendix D swaps the future observation only for an AUROC diagnostic; the method trained in Tables 1-3 always uses the realized observation in the Full view. Because q_{i,k,t}=tanh(e_{i,k,t}/2) is a bounded nonlinear function of the residual, a large part of the signal could come from the contrast between the two replay contexts, or from the specific placeholder phrase, rather than from the content of the actual observation. A matched random-observation training control would settle whether the gains in the main tables are due to observation calibration or to the extra replay-scoring pass and scaffold contrast. If random-observation OCSD matches OCSD, the 'observation-calibrated' explanation is unsupported; if it degrades to GRPO, this concern is resolved. This is a sharper form of the reader's worry about structural matching: the ablated view may not be a true counterfactual, and a training-time swap directly probes whether the true observation is the operative variable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses a confounding effect in privileged replay scoring for LLM agents: when future environment observations are replayed as privileged evidence, the resulting token-level support (log-probability change relative to the student view) conflates changes caused by the observation content with changes caused by the replay scaffold itself. The authors propose Observation-Calibrated Self-Distillation (OCSD), which constructs two structurally matched replay views—Full and Observation-Ablated—that differ only in whether the actual future observation is present. The difference of the two support terms (Eq. 6) is a parameter-free observation residual, mapped through tanh (Eq. 8) to modulate the token-level update strength of GRPO at high-NLL steps while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA with Qwen3-1.7B/4B/8B report consistent improvements over GRPO and several self-distillation baselines, and diagnostic analyses (AUROC against local environment feedback, word-level analyses, observation-swapping sensitivity) support the claim that the residual is better aligned with local feedback than Full-view support.","tokens_in":21981,"tokens_out":4860,"duration_ms":55549,"significance":"If the empirical claims hold, OCSD is a simple and well-motivated contribution to agentic RL: it identifies a real attribution problem in privileged replay scoring and proposes a clean, parameter-free contrast that cancels the student term and discounts shared scaffold effects. The paper is careful in several respects: the residual derivation is transparent, the architecture ablations are informative, the observation-swapping diagnostic is a good sensitivity check, and the code is publicly available. The main weakness is that the central empirical claim—that the training gains come specifically from the true future observation—is not directly tested at training time, and many of the reported improvements over strong baselines are within one standard deviation with only three seeds. The diagnostic AUROC is also computed only on the selected high-NLL steps used for training, leaving open the possibility that the apparent alignment is an artifact of step selection.","major_comments":[{"comment":"The central claim is that the residual in Eq. (6) isolates observation-specific support and that using it in Eq. (11) improves performance because it aligns with local environment feedback. The paper never tests this dependence at training time. Appendix D swaps the future observation only for an AUROC diagnostic at fixed checkpoints; the method trained in Tables 1-3 always uses the realized observation in the Full view. Since q_{i,k,t}=tanh(e_{i,k,t}/2) is a bounded nonlinear function of the residual, a substantial part of the signal could come from the contrast between the two replay contexts, or from the specific placeholder phrase, rather than from the content of the actual observation. A matched training-time control—replacing the realized observation in the Full view with a random same-operator donor observation, as in Appendix D—would settle whether the observed gains require the true observation content. If random-observation OCSD matches OCSD, the 'observation-calibrated' explanation is unsupported; if it degrades toward GRPO, the concern is resolved. This is a load-bearing gap for the paper's main narrative.","section":"Section 4.4 and Appendix D"},{"comment":"Several of the reported improvements over the strongest baselines are within one standard deviation, and with only three seeds the claim that OCSD 'consistently outperforms' is not statistically established. For example, Table 1 (WebShop Success, Qwen3-1.7B): OCSD 54.4±1.2 vs. SDAR 53.4±1.0; Table 2 (SearchQA Avg, Qwen3-1.7B): OCSD 43.1±1.2 vs. GRPO+OPSD 42.6±0.4; Table 2 (SearchQA ID, Qwen3-8B): OCSD 43.6±0.2 is below RLSD's 45.6±0.5. The paper should report paired per-seed comparisons or significance tests, or at minimum explicitly identify which differences are outside the bootstrap/standard-deviation overlap. Without this, the headline claim of consistent gains across all tasks and scales is not fully supported.","section":"Tables 1 and 2"},{"comment":"The AUROC diagnostic that supports the claim that the residual 'aligns better with local environment feedback' is restricted to the top-20% high-NLL interaction steps used for training and is computed on the training checkpoints themselves. This in-sample selection procedure could inflate the apparent alignment, because the step-selection criterion and the residual are both derived from the same policy at the same checkpoints. The paper should report the same AUROC analysis on the unselected steps, or on held-out evaluation tasks, to rule out the possibility that the alignment is an artifact of the selection procedure. The 'ambiguous cases excluded' rule also needs a sensitivity check, since the labeling is based on deterministic templates that could correlate with the residual in unforeseen ways.","section":"Section 2.2, Figure 2(c)-(d), Appendix C.3-C.4"}],"minor_comments":[{"comment":"The KL coefficient is denoted β_KLD in Eq. (13) and β_KL in Table 4, while Eq. (11) uses β for the advantage modulation coefficient; the notation is confusing and should be made consistent.","section":"Eq. (13) and Table 4"},{"comment":"The word-level analysis and the local-feedback AUROC in Figure 2 are defined on the top-20% high-NLL steps, but the NLL uncertainty score is first introduced in Section 3.2 (Eq. 9); the diagnosis in Section 2.2 would be easier to follow if the step-selection criterion were defined before the analysis.","section":"Section 2.2 and Appendix C.3"},{"comment":"The runtime breakdown figure uses a broken y-axis but does not state the units; please add the unit (seconds per iteration) and label the data values or the axis clearly.","section":"Figure 6"},{"comment":"The Vanilla rows are reported without standard deviations, unlike the other methods; if the vanilla evaluation is single-run, that should be stated explicitly in the caption or in Appendix B.","section":"Tables 1, 2, 6"}],"recommendation":"major_revision","confidential_remarks":"The core idea is publishable if the missing training-time observation control is added and the statistical picture is clarified. My main concern is that the current empirical evidence, while encouraging, does not yet uniquely support the 'observation-calibrated' mechanism: a random-observation training control is needed to exclude the possibility that the gains come from the extra replay-scoring pass or the scaffold contrast rather than from the actual observation content. Also, several key comparisons in Tables 1-2 overlap within one standard deviation; paired per-seed analysis would materially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper gives the agentic RL self-distillation line a genuinely clean trick—contrast a Full replay view against one where the future observations are blanked, and use the difference as a token-level signal. The residual is parameter-free and the student term cancels, so the derivation is clean. The diagnostics (word-cloud, functional categories, AUROC) are careful and the ablation table is informative. The paper deserves a serious referee.\n\nThe soft spots are real but not disqualifying. The biggest one: the paper never runs a train-time control where the Full view's future observations are replaced by random ones. Appendix D does this only as an AUROC diagnostic, not as a training condition. So the central claim—that gains come from observation calibration rather than just from the dual-view contrast—is not directly tested. The w/o Ablated Teacher ablation shows the residual is better than full support, but it doesn't isolate the observation content. A matched random-observation training arm would settle it. If OCSD with random observations matches OCSD, the 'observation-calibrated' interpretation is unsupported; if it degrades to GRPO, the concern is closed.\n\nAlso, no significance tests are reported, and several headline comparisons overlap within one standard deviation (e.g., Qwen3-8B WebShop success: OCSD 78.1±0.8 vs SDAR 75.0±0.8 is suggestive, not decisive). The main diagnostic only looks at the top-20% high-NLL steps used for training, so it's partly circular. Hyperparameters ρ and β were tuned on a single setup (Qwen3-4B/ALFWorld) and fixed everywhere; the sensitivity curves are nice but don't show cross-task stability.\n\nNone of this undercuts the core mechanism—the residual is well-motivated and the diagnostics do show it tracks local feedback better than full support. It just means the empirical claim is 'credible but not completely established.' The paper is honest and well-written, and the related work is properly cited. The code repository is announced but not verifiable from the preprint, so I can't vouch for reproducibility yet.\n\nFor a peer-review recommendation: desk reject would be wrong. Send it to a capable reviewer; with a train-time random-observation control added, it would be a solid paper. If I had to decide today, I'd take it conditionally.","headline":"A clean residual for separating observation signal from replay scaffold in agentic self-distillation, but the training-time control that would validate the interpretation is missing.","tokens_in":22557,"tokens_out":3232,"would_cite":true,"duration_ms":31625,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that contrasting token-replay scores under Full and Observation-Ablated views isolates the effect of future observations from replay-scaffold artifacts, and that using this calibrated residual to modulate token-level GRPO…","keywords":["reinforcement learning","LLM agents","on-policy self-distillation","token-level credit assignment","GRPO","observation ablation","agentic RL","self-distillation calibration"],"falsifier":"Train with a third replay view that contains the realized future observations but omits the future-action scaffold; if OCSD's advantage over Full support persists when no scaffold is present, the claim that the gain comes from removing scaffold confounds would be falsified.","tokens_in":21548,"feed_emoji":"🤖","tokens_out":8811,"duration_ms":88684,"temperature":0.7,"pith_summary":"LLM agents trained by reinforcement learning usually receive one sparse reward per trajectory, so token-level credit is hard to assign. On-policy self-distillation tries to fill this gap by re-scoring generated tokens under a privileged replay that includes future environment observations, but the replay scaffold itself shifts token scores and confounds the signal. This paper proposes Observation-Calibrated Self-Distillation (OCSD), which subtracts the support from an observation-ablated replay view (a structurally identical view with 'Observation: not provided' in place of real observations) from the full replay support, isolating score changes attributable to the future observation itself. The residual modulates token-level GRPO updates at high-uncertainty steps while preserving the trajectory-level update direction. Across ALFWorld, WebShop, and Search-QA and three model scales, OCSD reports consistent gains over strong baselines, with diagnostics indicating the residual tracks local environment feedback better than raw replay support.","feed_headline":"Contrasting two replay views sharpens credit assignment in agent RL","feed_subtitle":"Subtracting an observation-ablated score isolates real feedback, beating strong baselines on three agent benchmarks.","key_machinery":"The load-bearing object is the observation residual $e_{i,k,t} = \\log\\pi^F(y_{i,k,t}\\mid h_{i,k,t}, E^F_{i,k}) - \\log\\pi^A(y_{i,k,t}\\mid h_{i,k,t}, E^A_{i,k})$, computed from two structurally matched replay views. The Full view appends the actual future observations plus a naturalized schema of the intervening future action; the Observation-Ablated view keeps the same field order, format, and action scaffold but replaces each future observation with the fixed phrase 'Observation: not provided.' Contrasting the two views cancels the student log-probability and any score changes produced by the shared scaffold, leaving a bounded calibration signal $q_{i,k,t}=\\tanh(e_{i,k,t}/2)$ that modulates token-level GRPO advantages as $\\hat{A}^{\\mathrm{OCSD}}_{i,k,t}=\\hat{A}_i[1+\\beta\\,\\mathrm{sgn}(\\hat{A}_i)q_{i,k,t}]$ on NLL-selected steps. The step-selection heuristic concentrates the modulation on the top-$\\rho$ highest-NLL interaction steps within each trajectory, and the sign-preserving factor ensures the trajectory-level update direction is never reversed.","core_discovery":"The central claim is that the observation residual $e_{i,k,t} = \\delta^F_{i,k,t} - \\delta^A_{i,k,t}$, the difference between token support under Full and Observation-Ablated replay views, discounts score changes shared by the replay scaffold and isolates a token-level signal attributable to the true future observation. Because both supports are measured against the same student prediction, the student term cancels and the residual reduces to $\\log \\pi^F(y_{i,k,t}\\mid \\cdot) - \\log \\pi^A(y_{i,k,t}\\mid \\cdot)$. OCSD maps this residual through $\\tanh(e/2)$ and uses it to scale the magnitude of the GRPO advantage at each token in the top-$\\rho$ highest-NLL steps, keeping the sign of the trajectory advantage fixed. The paper argues, and supports with experiments, that this calibrated credit is more consistent with local environment feedback, achieving an AUROC of 0.707 for distinguishing valid from invalid actions versus 0.654 for full replay support, and that swapping the realized observation for an unrelated one collapses the signal, showing the residual tracks observation-action correspondence rather than the mere presence of extra text. Consistently higher success rates and scores on three agent benchmarks at three model scales are reported as the practical consequence.","pith_inferences":["The same contrast trick could apply to other privileged contexts, such as tool outputs, expert demonstrations, or hindsight hints, wherever a structural ablation can in principle isolate the target information from formatting artifacts.","Because the residual is a scalar per token, it could serve as dense reward shaping or as an update gate in policy-gradient algorithms beyond GRPO.","The reliance on high-NLL step selection suggests uncertainty-based gating is doing real work; replacing NLL with a learned uncertainty estimator might transfer the gains to tasks where rollout NLL is poorly calibrated.","A direct test of the scaffold-confound account would compare OCSD against a third view that includes real observations but omits the future-action scaffold; if the gains persist without the scaffold, the residual would not be the sole cause."],"forward_implications":["OCSD outperforms GRPO, OPSD, GRPO+OPSD, RLSD, and SDAR on overall ALFWorld success, WebShop success, and Search-QA average Exact Match across Qwen3-1.7B, Qwen3-4B, and Qwen3-8B.","The gains extend to out-of-distribution Search-QA datasets, so the benefit is not confined to the training distributions.","Diagnostic AUROC shows that the observation residual tracks local environment feedback more closely than either the Full or Observation-Ablated support alone.","Observation swapping degrades the residual's discriminative power to near chance, indicating the signal depends on genuine action-observation correspondence rather than on merely adding observation text.","OCSD adds only about 1.4% per-iteration runtime over GRPO, so the calibration is inexpensive to integrate."],"supporting_citations":[{"why":"Supplies GRPO, the trajectory-level reinforcement learning objective whose token-level advantages OCSD modulates.","marker":"Shao et al. 2024"},{"why":"Introduces the on-policy self-distillation replay-scoring paradigm that OCSD extends and corrects.","marker":"Zhao et al. 2026"},{"why":"Provides the ALFWorld benchmark used for training and evaluation.","marker":"Shridhar et al. 2020"},{"why":"Provides the WebShop benchmark and its score and success metrics.","marker":"Yao et al. 2022a"},{"why":"Provides the Search-QA benchmark with its seven datasets and search-augmented question-answering protocol.","marker":"Jin et al. 2025"},{"why":"RLSD is a direct baseline that also modulates reinforcement learning updates with self-distillation signals.","marker":"Yang et al. 2026a"},{"why":"SDAR is a direct baseline and the source of the unified training and evaluation protocol used across methods.","marker":"Lu et al. 2026"},{"why":"Provides the AUROC statistic used in the diagnostic tests showing the residual aligns with local environment feedback.","marker":"Hanley and McNeil 1982"},{"why":"Qwen3 is the base model family used at three scales in the experiments.","marker":"Yang et al. 2025"},{"why":"Motivates the high-NLL selective step heuristic that concentrates residual calibration on uncertain steps.","marker":"Wang et al. 2026c"}],"fun_headline_variants":["Observation-calibrated self-distillation isolates true token credit in RL","Subtracting ablated replay reveals observation-specific signal for agent credit","Contrasting replay views filters scaffold noise, boosts agent RL","OCSD: contrastive replay views separate real feedback from scaffold noise"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that substituting the fixed phrase 'Observation: not provided' for the real future observation removes exactly the observation-dependent information while leaving the model's handling of the replay format and future-action scaffold unchanged; if the mere presence of real observation text alters how the scaffold is processed, the residual still carries scaffold artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Observation-calibrated self-distillation isolates true token credit in RL","Subtracting ablated replay reveals observation-specific signal for agent credit","Contrasting replay views filters scaffold noise, boosts agent RL","OCSD: contrastive replay views separate real feedback from scaffold noise"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000764,"raw_usage":{"total_tokens":3453,"prompt_tokens":1076,"completion_tokens":2377,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":692,"completion_tokens_details":{"reasoning_tokens":2316}},"tokens_in":692,"tokens_out":2377,"duration_ms":18092,"temperature":1.0,"reasoning_tokens":2316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:33:02.197422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train with a third replay view that contains the realized future observations but omits the future-action scaffold; if OCSD's advantage over Full support persists when no scaffold is present, the claim that the gain comes from removing scaffold confounds would be falsified.","supporting_citations":[],"review_version":1}