{"id":"f3ebad11-a5df-4b39-8cd0-6fe636bc4d38","arxiv_id":"2608.05245","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"Search2Skill uses rubric-based reinforcement learning to jointly optimize when to search, how to search, and how to distill web evidence into reusable skills, improving expert-domain QA accuracy.","lead":"Search2Skill trains an LLM agent to search the web when it hits a knowledge gap, distill the retrieved evidence into reusable skills, and store those skills for future questions. The paper reports consistent accuracy gains over search-only and trajectory-only baselines on eight expert-level QA domains.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reuse reward rests on unvalidated teacher relabeling: if Qwen3.7-Max labels the K=3 similar questions incorrectly, RL can reward skills that fit those labels rather than true gold answers.","rationale":"The reader's weakest_assumption correctly flags the LLM-judge dependence, but the more precise soft spot is the reuse score, which is the ostensibly objective component of the reward. It is computed on teacher-relabeled neighbors, and the teacher is the same model that supplies the rubric judgments, so the 'independent' execution signal is not truly independent. This is load-bearing because s_reuse determines the magnitude of the decision bonus q(tau) in Eq. 4, and it is the only part of that bonus tied to measured downstream accuracy; if its labels are wrong, the whole rubric reward can be miscalibrated in a way that is not visible in the final exact-match evaluation unless the held-out domains happen to expose it. That said, the paper has substantial independent support: the held-out protocol disables search at test time, every domain-model combination improves over Direct Inference in both protocols, the raw-evidence comparison in Table 3 isolates abstraction, the cross-scale transfer in Table 5 is a meaningful generalization check, and Appendix F reports run-to-run variance for the streaming results. These make the central claim credible. The concern therefore does not overturn the verdict; it strengthens the case for the conditional requirement that the authors release the teacher-relabeled data and validate label fidelity before the numbers are treated as fully reliable.","tokens_in":25977,"tokens_out":17764,"duration_ms":184068,"concrete_test":"Sample 200 (anchor, similar-question, generated-skill) triples from RL rollouts; have Qwen3.7-Max relabel the similar questions as in Appendix B.2 and compare those labels against the original WebInstruct-verified gold answers plus an independent domain expert. Pre-register a teacher label accuracy threshold (e.g., 95%); if accuracy falls below it, retrain with corrected labels and re-run the held-out protocol. If the held-out deltas (+5.1% on Qwen3-4B, +6.6% on Qwen3-8B) shift by more than the three-run standard error, the concern lands. Additionally, re-run RL using an alternate teacher model (e.g., DeepSeek-V3.2) for similar-question relabeling and verify that the streaming and held-out gains remain within noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is not the rubric judge alone, but the execution-based reuse score s_reuse (Eq. 9, Appendix C.2), which is the only non-judge component of the decision reward. During RL, s_reuse is measured by injecting the generated skill into K=3 similar questions whose ground-truth answers are relabeled by Qwen3.7-Max, the same model that provides the grounding and query-quality judgments. The paper reports no validation of these relabeled labels, no agreement statistics, and no comparison against the original WebInstruct-verified answers. If the teacher's relabeling is noisy or systematically biased toward its own preferences, the policy can receive high reuse rewards for skills that improve performance on the teacher's labels without improving true accuracy. The exact-match held-out evaluation is the only gold-label checkpoint and is reassuring, but it is not a substitute for validating the training signal: a biased reuse reward could still produce a policy whose skills look good on the five training domains yet transfer less reliably to the OOD evaluation domains, or whose reported gains depend on the specific teacher model. Since s_reuse enters every exploring rollout's reward with weight lambda_c = 0.3, mislabeled reuse examples directly shape the policy gradient and could silently distort the learned search and skill-generation behavior.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Search2Skill, a framework that lets an LLM agent identify its own capability gaps, search external sources, and distill the retrieved evidence into persistent, reusable skills, trained end-to-end with a rubric-based reinforcement learning objective. The reward decomposes into an exploration-necessity term, a search-quality judge, and a skill-generation judge, gated by group-contrastive accuracy differences. Experiments on eight domains from SuperGPQA, MMLU-Pro, and EvoAgentBench report average accuracy gains over direct inference of 8.3% and 9.3% on Qwen3-4B/8B under a streaming protocol, and 5.1% and 6.6% under a held-out protocol, with additional analyses claiming that skill abstraction rather than raw evidence drives the gains and that the mined skills transfer across model scales.","tokens_in":26289,"tokens_out":5513,"duration_ms":54286,"significance":"If the results hold, Search2Skill is a valuable step toward self-evolving agents that genuinely extend their parametric knowledge boundary rather than only reorganizing internal experience. The paper is unusually detailed in its appendices: full prompt templates for the decision, exploration, and skill-only execution phases, explicit reward formulas, hyperparameters, and streaming standard deviations are all provided. The comparisons are also fairer than is common in this area: EvolveR and the search agent are re-trained under the paper's setup, and the held-out protocol disables search at test time. The main empirical pattern is internally consistent, and the cross-scale transfer result is interesting. However, the central training signal depends on an LLM judge and teacher-relabeled labels whose validity is not established, and a key mechanism claim about abstraction is supported by a confounded comparison. With those points addressed, the paper would merit publication.","major_comments":[{"comment":"The reuse score s_reuse is computed on K=3 similar questions whose ground-truth answers are relabeled by Qwen3.7-Max, the same model that provides the query-quality and grounding judgments. The paper reports no agreement statistics between these relabeled labels and the original WebInstruct-verified answers, nor any comparison against human annotations or an independent objective. Since s_reuse enters every exploring rollout's reward with weight lambda_c=0.3, a systematically biased teacher can shape the policy gradient toward skills that match the teacher's preferences rather than true correctness. The exact-match held-out evaluation is reassuring, but it validates the final policy, not the reward signal; a biased reward could still produce a policy whose reported gains are fragile or dependent on the particular teacher model. I ask the authors to report label-agreement statistics on a sample of relabeled questions and, ideally, to run an ablation that computes s_reuse against the original labels or an independent teacher.","section":"Appendix B.2 and Appendix C.2, Eq. (9)"},{"comment":"The held-out results, which are the core evidence for intrinsic skill reusability, are reported only as point estimates. Appendix F gives mean ± standard deviation for the streaming protocol on Qwen3-8B, but not for the held-out protocol, the EvoAgentBench results, or the Qwen3-4B runs. Since several held-out differences are modest (for example, Search2Skilltrain versus Memp on Qwen3-4B is +3.1 on average, with per-domain deltas sometimes negative), and LiveCodeBench has only 39 test tasks, the claim that Search2Skill 'consistently outperforms' baselines under the held-out protocol cannot be statistically assessed. I ask for error bars or confidence intervals over the collection splits and decoding seeds for Tables 2, 3, and 5.","section":"Table 2 and Appendix F"},{"comment":"The comparison between 'raw retrieved evidence' and 'abstracted skill' is confounded by presentation format. The raw condition stores and injects retrieved snippets, presumably with their original length and noise, while the abstracted condition provides a structured workflow and runnable Python functions. The 4.5% margin could therefore reflect formatting, prompt layout, or code-execution benefits rather than abstraction per se. Because the paper explicitly concludes that 'gains arise from skill abstraction rather than raw retrieved evidence,' this comparison needs a control that presents raw evidence through the same structured prompt and code interface, or otherwise manipulates only the abstraction step.","section":"Held-Out Evaluation, Table 3"}],"minor_comments":[{"comment":"The sentence 'Let ¯y expand ¯ydirect be the mean accuracies of Gexp and Gdirect' appears to be missing a conjunction; it should read 'Let ¯y_exp and ¯y_direct be the mean accuracies...'.","section":"Appendix C.1"},{"comment":"The variance table covers only Qwen3-8B streaming results; the main text should state whether the Qwen3-4B streaming results show similar run-to-run variance, or the corresponding table should be added.","section":"Appendix F"},{"comment":"The GitHub link in the abstract (ATH-MaaS/Marco-DeepResearch) looks like a placeholder or unrelated project URL rather than a code release repository; the authors should provide the actual anonymized repository or remove the link until code is available.","section":"Abstract"},{"comment":"The per-method delta column (e.g., '+0.7' for Search Agent) is not defined in the caption; it should be explicitly stated as the improvement over Direct Inference.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Search2Skill is a real step forward for self-evolving agents: it trains one policy to decide when to search, how to query, and how to distill retrieved evidence into reusable skills, and it shows transfer across model scales and held-out reuse. That combination is new relative to the cited work, and the experiments back it up. The appendices are unusually honest: full prompts, hyperparameters, trajectory examples, failure-pattern definitions, and streaming variance. The 4.5% margin of skill abstraction over raw evidence caching is a clean result that supports the core mechanism.\n\nThe soft spots are real but not fatal. The training reward leans on Qwen3.7-Max as teacher and judge: the reuse score s_reuse is computed on K=3 similar questions whose ground truths are relabeled by that same model, and Rbase on anchor questions uses teacher-produced answers. There is no human validation, no agreement statistics, and no comparison against the original WebInstruct-verified answers. If those labels are biased, the policy can be rewarded for skills that fit the teacher's preferences rather than true gold answers. This is a legitimate concern, and the stress-test note gets it right that s_reuse enters every exploring rollout with weight 0.3. I would not call it load-bearing, though: the held-out evaluation uses official gold answers, and the main gains appear there too. A badly miscalibrated training reward would more likely shrink than inflate that effect, but we cannot be sure without judge validation.\n\nTwo smaller issues: held-out tables have no error bars (streaming does), and code is promised but not released. The 'consistently outperforms' phrasing papers over one or two per-domain exceptions, but the text acknowledges them.\n\nWho is this for? Anyone working on agent memory, skill libraries, or self-evolution. It deserves a serious referee. If I were the editor I would send it out with a request for code/checkpoints, judge validation against human labels, and variance estimates for held-out runs. The central claim holds up for now, and the paper is a useful contribution either way.","headline":"Search2Skill is a genuine new combination—joint RL over when to search, how to search, and how to distill skills—with solid held-out evidence, though the training reward's dependence on a single LLM teacher/judge is the main thing to probe in review.","tokens_in":26863,"tokens_out":2561,"would_cite":true,"duration_ms":25696,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Search2Skill claims an LLM agent can push past its parametric knowledge by searching external sources and distilling the evidence into reusable skills, using a rubric-based reinforcement learning objective that jointly optimizes when to…","keywords":["search-driven skill distillation","rubric-based reinforcement learning","self-evolving agents","capability boundary","skill reuse","LLM agents","GRPO","web augmentation"],"falsifier":"Retrain Search2Skill exactly as described but replace the LLM judge's rubric scores with human-annotated ratings on the same dimensions for a random subset of training rollouts; if gains over direct inference shrink or disappear under human-rated rewards, the claim that rubric-RL with this judge drives the improvement is falsified. A simpler check: take a trained policy, disable the reuse-score component of the skill reward, and see whether held-out accuracy still rises; the paper's own ablation already suggests it would not, but only a direct comparison on a fresh domain settles it.","tokens_in":25760,"feed_emoji":"🔍","tokens_out":8140,"duration_ms":67933,"temperature":0.7,"pith_summary":"This paper tries to establish that a language agent can continue to grow beyond what its parameters encode by searching the web when it detects a gap in its own knowledge, and by distilling what it finds into structured, reusable skills. The proposed method, Search2Skill, trains a policy with a rubric-based reinforcement learning objective that rewards calibrated exploration decisions, high-quality search queries, and skills that are both faithful to the retrieved evidence and reusable on similar questions. Across eight expert-level domains, the trained policy improves over direct inference by 8.3% on a 4B model and 9.3% on an 8B model in a streaming setup, and by 5.1% and 6.6% when the skill library is reused on held-out questions with search disabled. The paper also shows that the gains come from skill abstraction rather than from caching raw retrieved evidence, and that a skill library mined by one model helps both smaller and larger executors.","feed_headline":"Distilled web search skills beat direct LLM answers by 9.3%","feed_subtitle":"A rubric-based RL scheme trains agents to search only when needed and turn evidence into reusable skills.","key_machinery":"The load-bearing object is the rubric-based reinforcement learning objective, composed of a task reward and three decision rewards. The exploration-necessity reward is a group-contrastive gate: within a batch of rollouts, exploration earns a quality bonus only when searching clearly outperforms direct answering, and direct answering earns a bonus when searching clearly hurts. The search-quality reward is an LLM-judged score of query abstraction and evidence gain, applied as a penalty on the bonus. The skill-generation reward combines a reuse score, measured by executing the skill on similar relabeled questions, with an LLM-judged grounding score that audits the skill's claims against the retrieved evidence. The final reward is a weighted, clipped combination of the task term and the gated quality bonus, trained with GRPO.","core_discovery":"The central claim is that external search, when supervised by the right reward signal, can be converted into persistent procedural knowledge that pushes an agent's effective capability boundary outward. The paper argues that this conversion requires optimizing three decisions jointly: when to search, how to search, and how to distill, and that a single outcome reward is too coarse for this credit-assignment problem. Its experiments show that a rubric-based reward with three complementary terms fixes the main failure modes of an untrained pipeline, that the distillation step contributes more than simply storing retrieved content, and that the resulting skills are not tied to the collecting model. If this is right, self-evolving agents need not be bounded by their priors: they can acquire expert-domain conventions and standard procedures that their parameters never contained.","pith_inferences":["A natural next test is whether the judge-based reward could be replaced by a learned reward model trained on human preference data; the paper does not validate its judge against human raters, so the faithfulness of the rubric scores remains an open empirical question.","The group-contrastive gating depends on both exploring and direct rollouts being sampled per group; if one strategy quickly dominates, the gate may stop providing signal, so the margin parameter $\\tau_0$ likely needs adaptive scheduling in settings with larger search budgets.","Because skills transfer across model scales, a practical extension is to decouple collection and execution: a small, cheap model mines the library while a larger model consumes it, lowering the cost of self-evolution.","The skill library is capped at 300 entries with no merging; adding deduplication or hierarchical organization could further compound the reuse gains reported here."],"forward_implications":["With the trained policy, a 4B model using skills reaches 56.5% on the Science domain, surpassing the unassisted 8B model at 50.0%.","A skill library mined by the 8B collector improves a 4B executor by +4.1% and a 14B executor by +3.5% over direct inference, so structured skills act as an external capability scaffold.","Removing any of the three rubric rewards degrades accuracy, with the exploration-necessity term most important: without it the search rate surges to 96.6% and accuracy drops by 3.8%.","RL reduces the average failure share across all five failure patterns from 25.3% to 11.9%, indicating the training fixes the whole pipeline rather than one stage.","Search2Skill acquires useful skills more efficiently than inward-looking baselines: on MMLU-Pro Philosophy it jumps to 57.2% after 168 questions, while memory-based baselines remain stagnant."],"supporting_citations":[{"why":"Supplies the rubric-reward idea that underlies the three decision rewards.","marker":"Gunjal et al. 2025"},{"why":"Provides evolving-rubric RL for deep research that the reward design builds on.","marker":"Shao et al. 2026"},{"why":"Search-R1 is the search-augmented agent baseline that isolates raw search capability.","marker":"Jin et al. 2025"},{"why":"Trace2Skill is the trajectory-only skill distillation baseline that must be beaten.","marker":"Ni et al. 2026"},{"why":"SkillOpt is the trajectory-based skill optimization baseline in the held-out comparison.","marker":"Yang et al. 2026"},{"why":"Supplies the streaming evaluation protocol and the reuse-score idea for skill generalizability.","marker":"Ye et al. 2026"},{"why":"SuperGPQA provides three of the eight expert-level evaluation domains.","marker":"Du et al. 2026"},{"why":"MMLU-Pro provides three of the evaluation domains and the open-ended QA setting.","marker":"Wang et al. 2024"}],"fun_headline_variants":["Rubric-based RL distills web search into reusable LLM skills","Web search skills boost LLM agents by 9.3% via RL distillation","Beyond priors: search2skill acquires expert procedures from the web","When to search and how: rubric RL turns web evidence into skills"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reward signal rests on the assumption that the LLM judge's scores for query quality and skill grounding, together with the reuse score computed on a few relabeled similar questions, correctly measure whether a distilled skill will help on future questions; if those measures are miscalibrated, the policy could learn to produce skills that look good to the judge but do not genuinely generalize.","fun_headline_variants_meta":{"raw":{"variants":["Rubric-based RL distills web search into reusable LLM skills","Web search skills boost LLM agents by 9.3% via RL distillation","Beyond priors: search2skill acquires expert procedures from the web","When to search and how: rubric RL turns web evidence into skills"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000337,"raw_usage":{"total_tokens":1839,"prompt_tokens":894,"completion_tokens":945,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":867}},"tokens_in":510,"tokens_out":945,"duration_ms":8789,"temperature":1.0,"reasoning_tokens":867,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T17:13:56.072828+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain Search2Skill exactly as described but replace the LLM judge's rubric scores with human-annotated ratings on the same dimensions for a random subset of training rollouts; if gains over direct inference shrink or disappear under human-rated rewards, the claim that rubric-RL with this judge drives the improvement is falsified. A simpler check: take a trained policy, disable the reuse-score component of the skill reward, and see whether held-out accuracy still rises; the paper's own ablation already suggests it would not, but only a direct comparison on a fresh domain settles it.","supporting_citations":[],"review_version":1}