{"id":"d226f10d-4de1-4ef6-a21b-6efd8e37a279","arxiv_id":"2608.05219","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Training LLM agents with reference-grounded self-distillation only at states matched to a successful trajectory improves task success on ALFWorld and WebShop over unconditionally applying the full reference.","lead":"This paper proposes State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), a training method that only applies privileged trajectory guidance when the agent's current state matches a state in a successful reference. It raises task success by 10 to 12 points over unconditional full-path distillation on ALFWorld and WebShop. A generalist reader may care because state-aware use of expert guidance is a reusable idea for training LLM agents.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"WebShop references are constructed oracle traces, not verified successful under the benchmark's own reward, so the state-matched routing premise (Eqs. 5-6) is unestablished on one of the two benchmarks.","rationale":"The reader's weakest assumption identifies the hand-engineered adapters' certification as load-bearing, especially the lack of validation beyond ALFWorld and WebShop and the WebShop oracle-trace concern. My stress-test converges on the same point but focuses it more sharply: the WebShop references are not merely unvalidated; they are explicitly acknowledged as potentially not successful under the benchmark's own scorer, directly contradicting the 'successful trajectory' premise used to define state-conditional certification in Eqs. 5-6. The paper's replay audit, which is the strongest evidence for matcher correctness, is performed only on ALFWorld. The WebShop matcher audit checks admissibility and set membership, not terminal success, so the mechanism's core assumption is untested on half the benchmark suite. This is not a fatal flaw: the ALFWorld validation, fixed-state interventions, routing ablations, and random-turn controls are genuine independent support, and the WebShop traces are deterministic and plausible, so the method may well work for the stated reason. But the paper's blanket 'consistently outperforms' claim extends to WebShop without the same certification evidence, making the central result conditional on an unverified assumption. The reader's CONDITIONAL verdict is therefore appropriate; I would not move it. A relatively simple replay audit on WebShop would settle the concern, so the condition is concrete and actionable. The secondary observation about Qwen2.5 WebShop Acc being flat reinforces the need for error bars and more careful 'consistency' language, but it is not the primary load-bearing issue.","tokens_in":21308,"tokens_out":4929,"duration_ms":54616,"concrete_test":"Run a WebShop executable-continuation replay audit analogous to ALFWorld Table 17. Sample a stratified set of matched turns from archived WebShop rollouts, restore the reached state (page type, current ASIN, selected options) from the realized action prefix, execute the matcher-selected candidate (unique action or full compatible set) followed by the remaining reference suffix, and record the WebShop environment's scored reward. Separately, execute each constructed 'oracle trace' in the environment from the initial state and record whether it achieves reward 1.0. If all replayed continuations reach the target purchase with score 1.0 and all reference traces are themselves successful, the certification concern is resolved; if replay success is below 100% or reference traces score below 1.0, the WebShop results are not attributable to the stated successful-trajectory mechanism.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SMRC-SD's gains come from withholding reference-conditioned distillation at unmatched states and grounding teacher context in the actually reached state, where the reference is a certified successful trajectory. That certification is explicit in Eqs. 5-6: each transition is tied to a demonstrated pre-action state from a successful path. On ALFWorld, the paper provides strong support via executable-continuation replay: 781/781 structured-state matches complete the candidate-plus-suffix successfully. On WebShop, however, no such verification exists. Section C states that WebShop references are 'deterministically constructed' traces from the target ASIN and requested options, called 'oracle traces' precisely because 'the bundled WebShop scorer can assign less than 1.0 even when a trace reaches the exact goal product and options.' Thus the reference paths are not certified as successful under the environment's reward, and the 'certified transition' premise of Eq. 6 is not satisfied on WebShop. The offline matcher audit in Table 11 only checks that candidates are admissible and that sampled actions belong to the rendered compatible set; it never executes the candidate plus the remaining suffix and measures terminal reward. This matters because the method's routing and contextualization depend on the reference containing a locally valid continuation. If WebShop references are plausible but not truly successful, the teacher's privileged context is not grounded in a demonstrated successful continuation, and the reported WebShop improvements (e.g., Qwen3 Acc 0.574 to 0.693) may be driven by a different mechanism than state-matched routing over certified successful trajectories. This leaves the central claim—that SMRC-SD consistently improves over unconditional successful full-path distillation—only partially evidenced: the mechanism is validated on ALFWorld but not on WebShop.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies state-reference mismatch in privileged on-policy self-distillation for multi-turn agents. It proposes SMRC-SD, which at each turn reconstructs compact state signatures from the student's execution and from a same-task successful reference, routes reference-conditioned self-distillation only to turns where the reached state matches a reference pre-action state and the reference action grounds in the current admissible set, and constructs the teacher context from the full successful path, a current-state summary, and the grounded candidate. The paper reports consistent gains over the unconditional FullPath-SD baseline on ALFWorld and WebShop for two model families, and supports the mechanism with fixed-state teacher interventions, routing and context ablations, matcher comparisons, and an executable-continuation replay on ALFWorld.","tokens_in":21645,"tokens_out":8230,"duration_ms":85901,"significance":"If the results hold, the paper makes a useful contribution by identifying a concrete failure mode of unconditional reference-conditioned distillation and by proposing a principled way to decide when a successful trajectory is locally valid for the state actually reached on policy. The fixed-state intervention design in Table 1 is a strong complement to end-to-end policy results, and the ALFWorld executable-continuation replay (781/781 structured-state matches) is convincing evidence that the matcher's matches correspond to executable continuations on that benchmark. The paper is honest about the training-only nature of the privileged components and releases code. The main concerns are the absence of uncertainty quantification for the headline policy comparisons and the lack of equivalent reference certification on WebShop.","major_comments":[{"comment":"The central claim that SMRC-SD consistently outperforms FullPath-SD is supported only by single-run point estimates. No confidence intervals, standard errors, or task-level bootstrap results are reported for Table 2 or for the ablations in Tables 3-5. With 128 tasks and four rollouts per task, a task-level bootstrap is straightforward and the paper already uses a game-cluster bootstrap for Table 1. Without this uncertainty quantification, the size and reliability of the gains cannot be assessed; for example, the Qwen2.5 WebShop Acc difference of 0.734 versus 0.736 is essentially a tie in point estimate, and the Qwen3 ALFWorld improvement of 0.746 to 0.865 needs a confidence interval to support the claim of consistent improvement. Please report per-task bootstrap intervals and, ideally, at least two independent seeds for the main configurations.","section":"Main Results, Table 2; Eqs. (4), (15)"},{"comment":"The conditional-transition premise in Eqs. (5)-(6) is that each reference transition is demonstrated from a pre-action state on a successful path. Section C concedes that the WebShop oracle traces are deterministically constructed and that the bundled WebShop scorer can assign less than 1.0 even when a trace reaches the exact goal product and options. The references are therefore not certified as successful under the environment's own reward, and on WebShop the matched/unmatched route is computed against plausible paths rather than demonstrated successful continuations. The offline audit in Table 11 verifies only that candidates are admissible and that sampled actions belong to the rendered compatible set; it never executes the candidate plus the remaining suffix and measures terminal reward, unlike the ALFWorld replay audit in Table 17. Please add a WebShop executable-continuation replay (student prefix plus candidate plus suffix, scored by the benchmark reward) or verify the constructed traces under the scorer; until then, the state-matched routing premise is validated on ALFWorld only.","section":"Supplementary C; Eqs. (5)-(6); Table 11"},{"comment":"The paper says it uses the official test splits and evaluates 128 tasks, but for WebShop the described protocol is a fixed 128-task subset of the first 500 held-out goals, not the full WebShop test set. The choice of 128 is not justified, and the phrase 'official test splits' is at least imprecise for WebShop. Since Table 2 is the headline evidence for the WebShop claim, please either evaluate the full 500-task held-out split, or justify the 128-task subset by citing prior work or by demonstrating that the subset is representative, and report per-task bootstrap intervals over it.","section":"Experimental Setup; Supplementary C"}],"minor_comments":[{"comment":"The raw candidate-minus-sampled action margins are negative for every teacher context, which may confuse readers. The paper correctly emphasizes the paired change relative to the no-privilege row, but a one-sentence clarification that the raw margins embed the fixed reasoning produced for the sampled action would prevent misinterpretation.","section":"Supplementary B, Table 8"},{"comment":"The WebShop audit reports 100.0% matching on search-home turns. Because a search action is almost always available at that page type, it would be more informative to also report the distribution of matched candidate types (query, option click, buy) for the other page types, so that the reader can see what the matcher actually certifies.","section":"Supplementary D, Table 11"},{"comment":"Figure 3 reports training dynamics from a single Qwen3 ALFWorld run. If the checkpoint and seed are the same as in Table 2, please state this explicitly in the caption; otherwise the dynamics plot should be labeled as a representative run rather than the main result.","section":"Main Results, Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is careful in its interpretive limits and the ALFWorld replay evidence is a strong positive. The main risks are the unverified WebShop oracle references and the lack of statistical reporting for the headline comparisons; both are fixable within the manuscript's scope. I would also ask the editor to confirm that the key recent baselines cited (Skill-SD, SDAR) are publicly available and that the reproduced numbers are from aligned settings."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on arXiv:2608.05219. The core idea is real: prior selection methods for privileged distillation regulate the teacher signal by confidence, outcome, or rollout structure, but nobody checks whether the reference trajectory actually contains a state compatible with the student's reached state. SMRC-SD does that with hand-engineered signatures, and the fixed-state interventions (Table 1) give clean evidence that FullPath-SD helps matched turns and hurts unmatched ones. The ablations in Tables 3-5 support both routing and contextualization as separate contributors. The replay audit on ALFWorld is the strongest piece: 781/781 structured-state matches re-execute the candidate plus the remaining suffix to success, and the same-count random-turn control rules out sparsity as the explanation. The paper is also unusually honest about its interpretive limits (Section J).\n\nNow the soft spots. The WebShop references are 'oracle traces' deterministically constructed from the target ASIN and options, not verified as successful under the benchmark's own reward. The paper explicitly says the bundled scorer can give less than 1.0 even for a perfect trace. That undermines the certified-transition premise of Eqs. 5-6 on WebShop. There's no executable-continuation replay on WebShop—only an offline matcher audit—so the mechanism that works so cleanly on ALFWorld is not demonstrated on the second benchmark. This doesn't break the paper, but it does mean the headline 'consistently outperforms' is only fully supported on ALFWorld.\n\nThe other issues are more minor. The main results have no confidence intervals, the 128-task evaluation isn't justified against standard splits, and everything is a single seed. These are addressable. The hand-engineered matching rules are environment-specific and could be a fragility, but the paper is upfront about that and provides a common interface.\n\nNet: this is a solid, novel contribution with careful evidence on ALFWorld and a genuine gap on WebShop. It deserves peer review, and I would expect the reviewers to ask for CIs, a WebShop replay audit, and a bit more justification of the eval split. I'd cite the state-matching idea. Not a desk reject.","headline":"A genuinely new routing mechanism for privileged self-distillation with strong ALFWorld evidence, but the WebShop half of the claim rests on unverified reference traces.","tokens_in":22161,"tokens_out":2600,"would_cite":true,"duration_ms":24412,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Privileged reference guidance is trustworthy only at states the reference actually covers; SMRC-SD routes self-distillation to matched states and grounds teacher context there, improving ALFWorld Average@4 from 0.746 to 0.865 and WebShop…","keywords":["state-matched routing","self-distillation","multi-turn agents","privileged guidance","state-reference mismatch","ALFWorld","WebShop","on-policy learning"],"falsifier":"Reuse the published adapters on a new interactive benchmark with different state semantics and compare SMRC-SD against FullPath-SD; if the matched candidate-plus-suffix replay success rate drops materially below the 100% reported on ALFWorld or the policy gain disappears, the support relation is not transferring. Likewise, score the 6,910 WebShop oracle traces with the environment's own WebShop scorer: if a substantial share of the traces used as references score below 1.0, one can test whether restricting references to scorer-verified successes changes the gains.","tokens_in":21121,"feed_emoji":"🤖","tokens_out":6719,"duration_ms":60633,"temperature":0.7,"pith_summary":"Privileged on-policy distillation gives a multi-turn agent dense training signal by letting a synchronized teacher re-score the student's own responses while conditioning on a successful reference trajectory. The problem is that the student's actions can carry it to execution states the reference never visits, so the same reference that is task-correct can still be locally misleading at exactly the turns where it is used. The paper proposes State-Matched Routing and Contextualized Self-Distillation (SMRC-SD), which first verifies whether the reached state is compatible with a supported pre-action state on the reference and only then applies distillation, constructing teacher context from the full path, a summary of the reached state, and the grounded candidate action. Across ALFWorld and WebShop with two model families, the method consistently outperforms unconditional full-path distillation; on Qwen3-1.7B it raises ALFWorld Average@4 from 0.746 to 0.865 and WebShop accuracy from 0.574 to 0.693. The paper's controlled ablations indicate that both matched-only routing and state-contextualized teacher guidance contribute, and that the routing benefit comes from matched-turn identity rather than merely from distilling on fewer turns.","feed_headline":"State-matched routing lifts agent success by up to 0.12","feed_subtitle":"Distilling only at states the expert path supports improves ALFWorld and WebShop success on both tested model families.","key_machinery":"The load-bearing object is the structured-state matcher: compact environment-specific state signatures $\\sigma_t$ for the on-policy reached state and $\\bar{\\sigma}_{g,k}$ for the reference pre-action states, reconstructed from task metadata and the action prefix, paired with the directional support relation $\\sigma_t \\models_g \\bar{\\sigma}_{g,k}$ and an action-grounding function $\\Gamma_g$ that checks whether the reference action can be performed from the current admissible set. The matcher selects the latest compatible position $k_t = \\max\\{k : C_t(k) = 1\\}$, sets the route mask $w_t$ to one only if such a position exists, and outputs the grounded candidate $\\tilde{a}_{t,k_t}$. This compatibility test determines both whether the privileged loss is applied and which continuation localizes the teacher context, so all downstream gains reported in the paper are attributed to the quality of this certification.","core_discovery":"The central claim is that an executable successful trajectory is a state-indexed resource, not a context to be applied unconditionally: each transition $\\bar{a}_{g,k}$ is certified only at its demonstrated pre-action state $\\bar{s}_{g,k}$, so when the on-policy rollout reaches a state $\\sigma_t$ that the reference does not support, conditioned self-distillation can actively lower the score of a valid action. SMRC-SD treats the reference as a conditional plan: a hand-engineered adapter reconstructs compact execution signatures, a directional support relation $\\sigma_t \\models_g \\bar{\\sigma}_{g,k}$ together with an admissible-action grounding check decides whether a match exists, and distillation is routed to matched turns only. On matched turns the teacher context is rendered from the full successful path, a one-line summary of the reached state, and the grounded candidate continuation. The reported result is that this jointly matched-and-routed procedure beats unconditional full-path distillation on both benchmarks and both model families, with the largest Qwen3-1.7B gain from 0.746 to 0.865 Average@4 on ALFWorld and 0.574 to 0.693 accuracy on WebShop. The ablations further show that random same-count turn selection does not reproduce the routing gain and that neither the state summary nor the candidate alone improves matched-turn guidance; the complete bundle is needed.","pith_inferences":["A natural extension is to replace the hard binary route with a confidence-weighted or learned compatibility score, which could capture partial matches the current hand-engineered relation rejects.","Because the adapters must be re-authored for each environment, the method's practical scope is the set of environments where such directional support relations can be specified; the paper demonstrates this for two benchmarks only.","The WebShop oracle traces are not required to score 1.0 under the environment's own scorer, so an untested variant is to require scorer-verified success as an additional routing condition and measure whether that changes policy quality.","The abstention design effectively turns the reference into a curriculum: the agent first learns on states it can certify, and one could test annealing the matching strictness during training to gradually admit harder states."],"forward_implications":["Unconditional full-path self-distillation should be replaced by state-matched routing in interactive environments, since applying the reference at unmatched states can actively reduce the probability of valid actions.","State-matched routing selects turns by execution-state identity; a random same-count control shows the gain is not from distilling on fewer turns.","The structured-state matcher retains 98.8% of history-based matches while adding coverage, and matched candidate-plus-suffix replay succeeds on all sampled structured-state matches.","The trained policy needs none of the privileged machinery at deployment: references, signatures, matcher, and teacher context are all removed.","Because routing abstains on unmatched turns, those turns still receive GRPO training, so the method changes how much privileged supervision enters the update without dropping the turn from on-policy learning."],"supporting_citations":[{"why":"Supplies the ALFWorld benchmark, its 3,553 expert walkthrough references, and the test-split evaluation protocol used for all ALFWorld results.","marker":"Shridhar et al. 2021"},{"why":"Supplies the WebShop benchmark and the oracle product-option traces from which WebShop references are constructed.","marker":"Yao et al. 2022"},{"why":"Defines the GRPO on-policy optimizer that trains every trajectory in the paper's loss.","marker":"Shao et al. 2024"},{"why":"Provides the Skill-SD baseline and the chosen-token K3 self-distillation estimator reused by SMRC-SD.","marker":"Wang et al. 2026a"},{"why":"Provides the SDAR gated self-distillation baseline and the SDAR-reported numbers SMRC-SD is compared against.","marker":"Lu et al. 2026"},{"why":"Establishes on-policy distillation as the supervision paradigm that privileged self-distillation extends.","marker":"Agarwal et al. 2024"}],"fun_headline_variants":["Match states before distilling: agent success up 0.12","State-gated distillation lifts success by up to 0.12","Distill only where states align — agents gain up to 0.12","Route distillation by state match — up to 0.12 gain","State-matched guidance beats unconditional distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method's transfer rests on the hand-engineered environment adapters and the directional support relation certifying state compatibility correctly: if the signatures misclassify a reached state, both the route mask and the grounded candidate used in training are wrong.","fun_headline_variants_meta":{"raw":{"variants":["Match states before distilling: agent success up 0.12","State-gated distillation lifts success by up to 0.12","Distill only where states align — agents gain up to 0.12","Route distillation by state match — up to 0.12 gain","State-matched guidance beats unconditional distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00152,"raw_usage":{"total_tokens":6201,"prompt_tokens":1170,"completion_tokens":5031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":786,"completion_tokens_details":{"reasoning_tokens":4945}},"tokens_in":786,"tokens_out":5031,"duration_ms":41000,"temperature":1.0,"reasoning_tokens":4945,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:50:51.182617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reuse the published adapters on a new interactive benchmark with different state semantics and compare SMRC-SD against FullPath-SD; if the matched candidate-plus-suffix replay success rate drops materially below the 100% reported on ALFWorld or the policy gain disappears, the support relation is not transferring. Likewise, score the 6,910 WebShop oracle traces with the environment's own WebShop scorer: if a substantial share of the traces used as references score below 1.0, one can test whether restricting references to scorer-verified successes changes the gains.","supporting_citations":[{"cited_title":"2022 , volume =","cited_arxiv_id":null,"evidence_quote":"Supplies the WebShop benchmark and the oracle product-option traces from which WebShop references are constructed."}],"review_version":1}