{"id":"9cbb198b-0bb0-4495-8752-eb38fe6fdfee","arxiv_id":"2607.17558","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Feedback-augmented self-distillation, training a search agent on its own successful rollouts, fails to improve retrieval-interleaved search agents: it collapses into generic, question-agnostic reasoning templates, and an EMA teacher only partially stabilizes it.","lead":"This paper tests whether a language model can improve its own web-searching by learning from its own successful answers, and finds the method fails because the model repeats generic reasoning templates that ignore the question. The authors diagnose the collapse and show that a slowly-updating teacher version of the model stabilizes training, though the result is still far below plain reinforcement learning.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Decoding collapse is asserted, not demonstrated: Figures 1 and 3 only show pass-rate dynamics and a definitionally coupled 'effective sample ratio,' so the central causal mechanism is inferred from the failure it is meant to explain.","rationale":"The paper's central claim is that feedback-augmented self-distillation fails because of decoding collapse driven by inconsistent supervision. This mechanism is the paper's main conceptual contribution and the basis for the proposed EMA/fixed-reference mitigations and the inconsistency decomposition. Yet the paper never directly measures template collapse; it infers it from pass-rate dynamics and a metric that is definitionally tied to pass rate. The paper's own admission that existing evaluation metrics would miss this phenomenon underscores the absence of a dedicated diagnostic. This is the single most load-bearing concern because if decoding collapse is not real, the entire explanatory story collapses into 'FA-SD just underperforms with these hyperparameters,' which would be a much weaker and less generalizable negative result. The concern is addressable with a direct metric, which the paper currently lacks. The reader's weakest_assumption identifies exactly this issue, so I agree. The verdict remains CONDITIONAL: the empirical negative result is credible and honestly reported (including the GRPO/OPD comparisons and the EMA improvement), but the mechanism is unverified. Requiring the direct collapse measurement as a condition for full acceptance is appropriate.","tokens_in":13816,"tokens_out":3896,"duration_ms":35272,"concrete_test":"Run FA-SD exactly as in §4.2 and, at checkpoints (e.g., steps 50/100/150/200) plus a GRPO control, decode N trajectories per question on a held-out set. Compute: (a) entropy over normalized first search-query templates; (b) average pairwise self-BLEU between trajectories for the same question vs different questions; (c) mutual information between question embeddings and generated reasoning/search-token embeddings. If decoding collapse is real, query-template entropy and question-trajectory MI should drop while cross-question self-BLEU approaches same-question values — and these trends should be absent or much weaker in the GRPO control. If the metrics stay flat or match the control, the negative result requires a different explanation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4.4 defines decoding collapse as 'models rely on recurring reasoning-and-search output templates, producing diverse-looking but input-question-agnostic trajectories.' The evidence offered is (i) near-zero pass rates in Figure 1 and (ii) a declining 'effective sample ratio' in Figure 3. But this ratio is the fraction of trajectories with a valid feedback signal — i.e., a group containing at least one successful rollout — so it is definitionally coupled to the pass rate and cannot independently diagnose template collapse. The paper states such behavior 'can be missed by existing evaluation metrics' but provides no metric that directly detects it: no query-template entropy, no cross-question trajectory similarity, no mutual information between the question and generated tokens, and no example of a template-collapsed trajectory. Thus the central causal chain — inconsistent supervision → decoding collapse → uninformative KL signal — is not tested. Alternative explanations (high-variance single-sample reverse-KL gradients over long interleaved rollouts, noise in success-rollout selection, or the 200-step budget for FA-SD versus 1000 steps for external-teacher OPD in Table 2) remain unruled-out. The later model-inconsistency versus prompt-inconsistency decomposition (§4.5–4.7) and the FA-MOPD interpretation rest on this same unmeasured collapse layer.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies Feedback-Augmented Self-Distillation (FA-SD) for retrieval-interleaved search agents. FA-SD uses successful rollouts from the current policy as privileged demonstrations to construct a feedback-augmented self-teacher and performs token-level KL-style distillation into the unconditioned student. Across seven open-domain QA benchmarks, the authors report that FA-SD does not sustain improvement: unclipped and PPO-style clipped variants show pass rates close to zero for most of training, with a transient rebound around step 180. They attribute this failure to 'decoding collapse,' a tendency to produce recurring reasoning-and-search templates that are input-question-agnostic, making the KL signal uninformative. They further decompose the instability into model inconsistency (drifting self-teacher) and prompt inconsistency (feedback conditioning), and show that a fixed reference or EMA teacher stabilizes training, with the EMA variant reaching 0.206 average EM versus 0.114 for the base model. They extend feedback augmentation to external-teacher MOPD and find it underperforms standard MOPD, which they interpret as additional evidence of prompt inconsistency.","tokens_in":14015,"tokens_out":4456,"duration_ms":40043,"significance":"If the empirical measurements are taken at face value, the paper provides a useful, honestly reported negative result on a timely question: whether self-distillation from privileged successful rollouts can provide dense supervision for long-horizon search agents. The multi-benchmark evaluation, the comparison against GRPO (0.348) and Qwen7B-Instruct (0.304) as strong upper reference points, and the systematic ablation of teacher regularization are concrete strengths. The EMA/fixed-reference comparison is a sensible diagnostic that supports the practical recommendation that self-teacher drift must be controlled. However, the paper's central causal claim—that decoding collapse is the mechanism behind FA-SD's failure—is not directly evidenced by the metrics presented. The effective sample ratio is definitionally coupled to the pass rate, and no trajectory-level diagnostic is provided. The scientific contribution would be stronger if the mechanism were treated as a hypothesis and the headline finding framed as a measured negative result with proposed explanations, rather than as an established failure mode.","major_comments":[{"comment":"The named mechanism 'decoding collapse' is asserted, not demonstrated. The evidence is (i) near-zero pass rates and (ii) a declining 'effective sample ratio' defined as the fraction of trajectories that provide a valid feedback signal—i.e., that come from a rollout group containing at least one successful rollout. This ratio is definitionally coupled to the pass rate, so it cannot independently diagnose template collapse. No metric directly tests input-question-agnostic template use: no question-conditioned trajectory similarity, no template entropy, no mutual information between question and generated tokens, and no example of a collapsed trajectory. The claim that such behavior 'can be missed by existing evaluation metrics' requires a metric that detects it; none is provided. This unmeasured interpretive layer underpins the causal chain in Section 4.7, so the main 'why' contribution re","section":"§4.4, Figures 1 and 3"},{"comment":"Alternative explanations for the observed dynamics are not ruled out. FA-SD is trained for only 200 steps while external-teacher OPD gets 1000 steps; the single-sample reverse-KL estimator in Eq. (4) has high variance over long interleaved rollouts; and success-rollout selection via exact match is noisy in multi-hop QA. The transient rebound around step 180 and the failure of the student to track the self-teacher (Figure 2) are equally consistent with optimization instability, variance, or insufficient budget as with template collapse. To make decoding collapse load-bearing, the paper should include a control (larger budget, multi-sample KL estimator, or a direct trajectory-similarity metric) or explicitly reframe the mechanism as a hypothesis.","section":"Table 2, Eq. (4), Figure 1"},{"comment":"The EMA-regularized FA-SD improves the average from 0.114 to 0.206, which conflicts with the title's blanket claim that FA-SD 'fails to improve.' The paper qualifies this as 'does not sustain improvement across variants,' but the title and abstract overstate the negative result. If the intended claim is that FA-SD is dominated by GRPO (0.348) and Qwen7B-Instruct (0.304), that is a different and defensible claim, but it should be stated as such. This framing issue affects how readers interpret the paper's central contribution.","section":"§4.5, Table 5, Title/Abstract"},{"comment":"The interpretation that FA-MOPD underperforms MOPD 'indicating that privileged feedback can introduce additional prompt inconsistency' is not supported by direct measurement. The degradation could be due to distribution shift in the teacher prompt, to conditioning on noisy successful rollouts, or to interaction between the feedback and the student's on-policy prefixes. Without ablating the feedback template (e.g., providing the same successful rollout to the student, or using oracle feedback), the decomposition into model versus prompt inconsistency remains speculative. Because this decomposition is part of the paper's explanation for FA-SD's failure, it needs additional empirical support.","section":"§4.6–4.7, Figure 5"}],"minor_comments":[{"comment":"The text 'identify and characterizedecoding collapse' contains a missing space and should read 'identify and characterize decoding collapse.'","section":"Section 1"},{"comment":"The model is referred to inconsistently as 'Qwen2.5-3B-Base' and 'Qwen3B-Base' in the tables and text; unify the terminology.","section":"Tables 5–6 and Appendix C.4"},{"comment":"The divergence D is not specified when the loss is introduced. Since the gradient derivation in Eq. (4) assumes reverse KL, state this when defining Eq. (3) to avoid ambiguity for readers.","section":"Equation (3)"},{"comment":"The 'effective sample ratio' is described qualitatively but never given a formula. A precise definition (e.g., fraction of rollout groups with at least one successful trajectory) would clarify why it is correlated with pass rate and help the reader evaluate whether it can serve as an independent diagnostic.","section":"Section 4.4"},{"comment":"The sentence 'We include this result to complement the training-dynamics analysis' appears in both C.3 and C.4 with identical wording; rephrase to avoid duplication, and provide a brief interpretation of the OPD-after-GRPO degradation rather than only reporting the numbers.","section":"Appendix C.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is an honest negative result with a useful empirical foundation. The main weakness is that the central causal mechanism—decoding collapse—is presented as established while the evidence only shows a definitionally coupled diagnostic. If the authors can add trajectory-level evidence (e.g., cross-question similarity, template detection, or a control experiment), the paper would be substantially stronger. If they cannot, they should reframe the contribution as a measured negative result with proposed hypotheses, and adjust the title accordingly. I would support publication after such revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline result is real: feedback-augmented self-distillation (FA-SD) does not improve a 3B search agent across seven QA benchmarks, and stabilizing the teacher with EMA gets you to 0.206 average EM versus 0.114 base but still far behind GRPO at 0.348. That negative measurement is worth having.\n\nWhat's new: the specific configuration—RDS-style self-distillation applied to retrieval-interleaved rollouts, with successful rollouts as privileged demonstrations—hasn't been tested like this before. The paper is also honest: it reports the dominating GRPO and Qwen7B baselines, admits EMA/fixed-reference only partially help, and shows feedback augmentation also hurts external-teacher MOPD. The failure to internalize prompt-conditioned behavior is a plausible and useful observation.\n\nThe soft spot is the causal story. 'Decoding collapse' is the paper's explanation, but it is never directly observed. The evidence is near-zero pass rates and an 'effective sample ratio' defined as the fraction of trajectories that contain at least one successful rollout—so it is definitionally tied to the pass rate and can't independently diagnose template collapse. The paper says existing metrics would miss the phenomenon but ships no metric that catches it. No trajectory examples of collapse, no cross-question similarity, no mutual-information check. So the reader's stress-test is on target: the mechanism is asserted, not demonstrated. Alternative explanations—single-sample reverse-KL noise over long rollouts, noisy success selection, or the 200-step budget for FA-SD versus 1000 for external-teacher OPD—are not ruled out. Also, the failing main variants never get final eval numbers, and there are no seeds or error bars. These are real, but they are fixable; the empirical direction is probably right.\n\nThis is a paper for people building self-improving search agents and anyone applying OPSD-style distillation to agentic tasks. It deserves a serious referee—the negative result is useful and the honesty is refreshing—but it needs a stronger evidence layer before the mechanism claim can be taken at face value. I'd send it out, with instructions to focus on whether decoding collapse can be directly measured.","headline":"A genuinely useful negative result about feedback-augmented self-distillation for search agents, but the named mechanism—'decoding collapse'—is inferred rather than demonstrated.","tokens_in":14645,"tokens_out":2209,"would_cite":true,"duration_ms":19548,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper establishes that feedback-augmented self-distillation fails to improve retrieval-interleaved search agents because models settle into input-agnostic decoding templates, and that stabilizing the self-teacher only partially fixes t","keywords":["self-distillation","retrieval-interleaved search agents","decoding collapse","on-policy distillation","teacher regularization","EMA teacher","open-domain QA","negative results"],"falsifier":"At a late FA-SD training step, take rollouts for many different questions and measure (1) token/query overlap across questions (e.g., self-BLEU or entity overlap) and (2) the KL loss between the unconditioned student and the feedback-augmented self-teacher. If trajectories remain clearly question-specific while pass rates hover near zero, the decoding-collapse explanation is falsified; if trajectories are near-identical across questions while the KL loss stays low, it is confirmed.","tokens_in":13527,"feed_emoji":"🔍","tokens_out":5320,"duration_ms":44563,"temperature":0.7,"pith_summary":"This paper tries to establish why feedback-augmented self-distillation (FA-SD), a training method that distills token-level behavior from a model prompted with a successful search rollout, does not improve retrieval-interleaved search agents. Across seven open-domain QA benchmarks, FA-SD variants fail to sustain improvement even though the feedback-augmented teacher branch is often stronger than the student. The paper's causal diagnosis is decoding collapse: trajectories look diverse but follow recurring reasoning-and-search templates that ignore the question, so the KL signal carries no useful information. Stabilizing the teacher with an exponential moving average partially rescues training—raising final average score from 0.114 for the base model to 0.206—but shows that self-teacher instability, especially prompt inconsistency, is a central obstacle. The work matters because it isolates a failure mode that aggregate metrics miss and locates the problem in supervision consistency rather than teacher capability.","feed_headline":"Self-distillation collapses into question-agnostic templates","feed_subtitle":"Feedback from successful searches helps at prompt time but fails to transfer; an EMA teacher only partly repairs the damage.","key_machinery":"The central object is the FA-SD loss, which distills token-level distributions from a feedback-augmented self-teacher—the same policy conditioned on a successful rollout from the rollout group—into the unconditioned student via reverse KL, with a stop-gradient on the teacher branch. The paper's named failure mode is decoding collapse, defined as recurring reasoning-and-search output templates that appear diverse but are agnostic to the input question, rendering the KL signal uninformative. The stabilizing machinery is teacher regularization: a fixed reference teacher (the student at the start of training) or an exponential moving average (EMA) teacher that slows parameter drift and makes sup","core_discovery":"The central claim is that feedback-augmented self-distillation (FA-SD) does not reliably transfer the advantage of seeing a successful rollout into the unconditioned search policy. In FA-SD, the same policy is prompted with a correct earlier attempt and used with a stop-gradient as a per-token teacher for the unconditioned student; the paper finds this supervision collapses: the model adopts recurring reasoning-and-search templates that produce diverse-looking but input-agnostic trajectories, making the reverse-KL distillation signal uninformative. The paper supports this with training dynamics (near-zero pass rates, declining effective sample ratio) and by showing the self-teacher branch ou","pith_inferences":["Beyond the paper: decoding collapse could be tested directly by measuring trajectory overlap across questions at a late training step—e.g., self-BLEU of search queries or entity overlap—rather than inferring collapse from pass rates alone.","Beyond the paper: the same inconsistency decomposition (model drift vs. prompt conditioning) likely applies to other self-improvement settings, not just search agents; EMA stabilization may be broadly useful whenever a model distills from its own conditioned outputs.","A plausible alternative explanation the paper leaves open is that single-sample reverse-KL gradients over long interleaved rollouts are too high-variance to train reliably; comparing multi-sample teacher estimates or lower-variance estimators could separate optimization noise from template collapse.","If decoding collapse is real, evaluation protocols for agentic RL should include input-specificity checks—such as requiring that search queries contain question-specific entities—so that diverse-looking but degenerate policies are not scored as successful."],"forward_implications":["If the central claim is correct, FA-SD without teacher regularization will not improve retrieval-interleaved search agents even when successful rollouts are available as privileged demonstrations.","EMA-regularized FA-SD can improve over the base model (0.206 vs 0.114 average score) but still falls far short of an RL-trained search agent (0.348), suggesting dense self-distillation alone is not a substitute for outcome-level reinforcement.","Prompt-time feedback can elicit stronger behavior from the same model, but that advantage is not reliably internalized by the unconditioned policy through token-level KL distillation.","Feedback augmentation does not straightforwardly generalize to external-teacher on-policy distillation: adding a successful rollout to the teacher's prompt (FA-MOPD) underperforms standard MOPD, indicating privileged feedback can introduce prompt inconsistency even with a strong teacher.","Training diagnostics should track the effective sample ratio—the fraction of rollouts providing a valid distillation signal—not just pass rate or final EM, because aggregate metrics can miss decoding collapse."],"fun_headline_variants":["Decoding collapse: Why self-distillation fails on search agents","Retrieval agents: self-distillation falls prey to template collapse","Why self-teaching collapses on search: Question-agnostic templates","Question-agnostic trajectories defeat feedback self-distillation","Why feedback self-distillation fails: decoding collapse"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the observed training collapse is caused by decoding collapse—recurring, input-agnostic templates—rather than by another property of the optimization (e.g., noisy single-sample reverse-KL gradients over long trajectories, noisy success labels, or the short 200-step training budget).","fun_headline_variants_meta":{"raw":{"variants":["Decoding collapse: Why self-distillation fails on search agents","Retrieval agents: self-distillation falls prey to template collapse","Why self-teaching collapses on search: Question-agnostic templates","Question-agnostic trajectories defeat feedback self-distillation","Why feedback self-distillation fails: decoding collapse"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":2989,"prompt_tokens":754,"completion_tokens":2235,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":2152}},"tokens_in":498,"tokens_out":2235,"duration_ms":13160,"temperature":1.0,"reasoning_tokens":2152,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T17:38:52.300301+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"At a late FA-SD training step, take rollouts for many different questions and measure (1) token/query overlap across questions (e.g., self-BLEU or entity overlap) and (2) the KL loss between the unconditioned student and the feedback-augmented self-teacher. If trajectories remain clearly question-specific while pass rates hover near zero, the decoding-collapse explanation is falsified; if trajectories are near-identical across questions while the KL loss stays low, it is confirmed.","supporting_citations":[],"review_version":1}