{"id":"bf4c147b-d391-4e8f-8bdf-ef25737cf2eb","arxiv_id":"2609.38024","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RASO retrieves and adapts existing agent skills from a public corpus, using them to initialize and iteratively update task skills, and beats retrieval-free skill optimizers on four benchmarks with two models.","lead":"This paper proposes a way to build and improve the instructions, called skills, that AI agents follow, by pulling relevant pieces from a large public library of existing skills and rewriting them to fit the target task and tools. The method reduced the need for expensive trial runs and improved scores on four agent benchmarks with two AI models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Residual test/validation leakage into the retrieval corpus is the load-bearing threat; the paper's own blocklist (§A.1) is unreleased and unmeasured, so the central claim that gains come from transferable cross-harness knowledge rather than memorized benchmark content is not yet falsifiably…","rationale":"The reader's weakest_assumption—residual leakage of evaluation-specific skills from GitSkills—is the single concern most likely to invalidate the paper's central claim that RASI and RASU gains come from transferable cross-harness procedural knowledge. I agree with this identification because the paper itself states in §A.1 that the corpus may contain benchmark-specific skills, and its only control is an unreleased, unmeasured keyword blocklist. The paper's other components are strong: the RASI-vs-RFSI isolation in §A.1 is a well-designed attempt to attribute the gain to retrieval, the ablations in Tables 2–4 cleanly separate initialization and update effects, and the matched-rollout and cost tables (Tables 9–10) directly address the practical-efficiency objection. I found no internal inconsistency, no unsupported cited theorem, and no fabricated evidence. The remaining load-bearing gap is entirely empirical: without a released blocklist and a residual-leakage audit, the central interpretation is not falsifiable. If the blocklist is released and a residual-leakage audit shows negligible contamination, the central claim would hold as stated; if not, the interpretation changes. Therefore the reader's CONDITIONAL verdict is appropriate, and I recommend no change.","tokens_in":19454,"tokens_out":2344,"duration_ms":18140,"concrete_test":"Request or independently reconstruct the per-benchmark blocklists and rerun RASI initialization (and one RASU iteration) on SpreadsheetBench and WebShop with two retrieval settings: (i) the full non-blocklisted corpus, and (ii) a corpus restricted to sections whose repository mentions neither the benchmark name nor its dataset name nor any known solution-hinting strings (e.g., answer headers, expected-output patterns). If the restriction removes sections that resemble target-solution code and the RASI/RASU margin over RFSI collapses or reverses, the central claim must be revised. A cheaper first check: release the blocklists and compute, per benchmark, the fraction of the retrieved top-5 sections that contain the benchmark or dataset name, reporting a residual-leakage statistic, and the performance deltas when those sections are removed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"RASI and RASU gains are claimed to reflect transferable procedural knowledge (e.g., the satellite-imagery boolean-mask lesson adapted to SpreadsheetBench in Fig. 5). For that claim to hold, retrieved sections must not encode evaluation-specific content. The authors themselves flag this in §A.1: \"Since the external skill corpus is collected from public repositories, it may contain skills written specifically for our evaluation benchmarks.\" Their sole control is a per-benchmark keyword blocklist that is not released, not measured for precision or recall, and not audited for residual leakage. Three consequences follow. First, repositories that do not name a benchmark but contain its tasks evade keyword matching, as the authors acknowledge when choosing repository-level exclusion. Second, no ablation restricts the corpus to only non-matching sections: Fig. 4 varies corpus size but keeps all non-blocklisted repositories, so it cannot detect residual contamination. Third, the RASI-vs-RFSI isolation in §A.1 is clean in design but depends entirely on the blocklist being complete; if it is not, the RASI/RASU margins (e.g., +5.61 on OfficeQA, +11.30 on WebShop for Qwen-3.5-9B) could come from retrieving target-solution snippets rather than from cross-harness adaptation. The threat is not that the cited public corpus is suspect, but that the central interpretation—\"adapting diverse external knowledge rather than relying on particular source documents\" (§1 contributions)—is currently indistinguishable from memorized benchmark solutions. This is a correctness and interpretation risk, not an ad hominem: the authors honestly acknowledge the potential contamination and state a mitigation; what is missing is a released blocklist and a residual-leakage measurement that would make the claim falsifiable. I am not attacking the cost claims (Tables 9–10) or the matched-rollout comparison (Table 10), which are well supported and orthogonal.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes Retrieval-Augmented Skill Optimization (RASO), which exploits an external skill corpus as prior knowledge at two stages: Retrieval-Augmented Skill Initialization (RASI), which builds an initial skill from task/harness descriptions plus retrieved-and-adapted lessons without any rollouts, and Retrieval-Augmented Skill Update (RASU), which iteratively refines the skill by retrieving external knowledge guided by rollout feedback. A shared Cross-Harness Adaptation operation rewrites retrieved sections into the vocabulary and tool set of the target harness, addressing domain and harness mismatch. The paper evaluates RASO on OfficeQA, SpreadsheetBench, ALFWorld, and WebShop with GPT-5.6-Luna and Qwen-3.5-9B, reporting that RASI beats retrieval-free initialization (RFSI) at zero rollouts and that RASO beats TextGrad, GEPA, SkillOpt, and WikiSkill at equal or lower rollout counts (Tables 1, 9, 10). The central interpretive claim is that gains come from adapting diverse external procedural knowledge rather than from benchmark-specific or directly matching source documents.","tokens_in":19944,"tokens_out":6147,"duration_ms":48176,"significance":"If the claimed results hold, RASO makes a useful empirical contribution: it shows that large public skill corpora can be exploited cheaply throughout skill optimization without extra rollouts, and the component-wise evidence is reasonably strong. The paper's strengths include a clean RFSI-vs-RASI isolation (identical task/harness descriptions, procedure decomposition, and system prompt, differing only in the grounded lessons), fixed-initialization comparisons that separate the update contribution (Tables 4, 6-8), matched-or-fewer rollout counts (Table 10), explicit cost accounting (Table 9), and a concrete qualitative case of genuine cross-domain transfer from satellite-imagery code to a spreadsheet task (Fig. 5). The main reservations concern the unreleased and unmeasured corpus-exclusion control, the unstated split used for the K-selection analysis, and small margins on some benchmarks relative to three-seed standard errors; all are addressable within the manuscript's scope.","major_comments":[{"comment":"The load-bearing control for the claim that RASO's gains reflect transferable cross-harness knowledge is the per-benchmark keyword blocklist in §A.1, which is neither released nor evaluated for precision or recall. Because the central interpretation of the RASI/RASU margins (e.g., +11.30 on WebShop for Qwen-3.5-9B, Table 1) is that they come from adapting diverse external procedural knowledge rather than from retrieved target-solution content, the authors should release the blocklist, quantify residual leakage into the retrieval index (e.g., by sampling non-blocklisted repositories for benchmark-specific tasks), and ideally re-run the RASI-vs-RFSI isolation with the corpus restricted to sections explicitly flagged as non-matching. The corpus-size ablation in Fig. 4 varies the fraction of repositories while keeping all non-blocklisted ones, so it cannot detect residual contamination; the authors acknowledge this risk in sentence [b245] but do not measure it.","section":"§A.1, Fig. 4, Table 1","authors_acknowledge":"Since the external skill corpus is collected from public repositories, it may contain skills written specifically for our evaluation benchmarks.","authors_acknowledge_p":0.87},{"comment":"The choice of K=5 is motivated by Figure 3, which plots performance against K, but the figure caption and the surrounding text do not state whether the curves report validation or test performance on OfficeQA and SpreadsheetBench. If K was selected using test-set results, then the main-table numbers inherit test-set-tuned hyperparameters and the reported margins are optimistic. The authors should state which split Figure 3 reports and, if it is the test split, either re-select K on D_val or report the sensitivity of the reported results to K across all four benchmarks.","section":"§5.2, Fig. 3","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"Several headline margins are small relative to the reported standard errors over only three seeds, e.g., ALFWorld 74.13±1.00 vs 72.64±0.90 and WebShop 46.61±0.11 vs 45.54±0.35 for GPT-5.6-Luna, and the paper reports no significance testing. The bold 'consistently outperforms' claim in the abstract and conclusion therefore rests more heavily than acknowledged on the large-margin benchmarks (SpreadsheetBench, OfficeQA, and WebShop with Qwen-3.5-9B). Reporting per-benchmark significance (e.g., paired bootstrap or a paired test across seeds) or explicitly flagging the margins that fall within one standard error would make the claim's strength transparent without contradicting the core results.","section":"§5.1, Table 1","authors_acknowledge":null,"authors_acknowledge_p":null}],"minor_comments":[{"comment":"The blocklist is described only as matching 'the benchmark name and the names of the datasets it is built on'; publishing the exact keyword list would let readers assess recall and reproduce the corpus construction.","section":"§A.1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"No code, prompts, or retrieval index are released. Given that the RASI/RFSI and RASU-vs-baseline comparisons hinge on prompt and retrieval details, a public release (including the blocklist) is needed for the empirical claims to be verifiable.","section":"Appendix","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The abstract's phrasing 'consistently outperforms baselines without retrieval-augmented skill initialization and updating' should name the baseline set (TextGrad, GEPA, SkillOpt, WikiSkill under RFSI initialization) to avoid over-generalization.","section":"Abstract","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The model labels (GPT-5.6-Luna, Qwen-3.5-9B) appear as the final row of each block rather than as grouped headers, making the table harder to read; restructuring the panel headers would aid comprehension.","section":"Table 1","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The statement that RASO remains 'within $0.06 of the cheapest method on SpreadsheetBench' should be stated with the exact numbers (0.49 vs 0.43 USD) for clarity.","section":"§A.5","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Colleague,\n\nTwo things to know. First, this is a genuinely useful method paper: putting retrieval-augmented initialization (RASI) and retrieval-augmented iterative update (RASU) together with cross-harness adaptation is a real combination that none of the cited works — SkillRouter, AgentKB, TextGrad, GEPA, SkillOpt, WikiSkill — operate jointly. Second, the one unclosed hole is retrieval-corpus contamination, and it is the load-bearing one.\n\nWhere it earns credit. The experimental hygiene is better than typical for this subfield. Rollout counts are matched across methods (Table 10), cost tables are reported honestly, and the fixed-initialization comparisons in the appendix (Tables 6-8) isolate the update procedure from the initialization. The RASI-vs-RFSI contrast in A.1 is clean by design: identical task/harness descriptions, identical procedure decomposition, identical initialization prompt — only retrieval and adaptation differ. The ablations in Table 2 show retrieval helps at both stages independently and additively. Updates are accepted only on held-out validation, K=5 was tuned on validation, and seed variance is reported. The authors openly state the contamination risk (b245: the corpus 'may contain skills written specifically for our evaluation benchmarks') and describe their repository-level blocklist.\n\nWhere it is soft. The blocklist is the whole mitigation and it is not yet verifiable: unreleased, manual, unmeasured for precision or recall, with no residual-leakage audit. Fig. 4 varies corpus fraction but keeps all non-blocklisted repositories, so it cannot detect residual leakage. That means the headline margins — especially +11.30 on WebShop for Qwen — are not yet distinguishable between transferable cross-harness procedural knowledge and retrieved benchmark-solution snippets. Because the authors chose repository-level exclusion specifically to catch skills that evade keyword matching, this is an honest, acknowledged risk; I do not read it as a credibility problem, but until the blocklist ships with a residual-leakage check, the central interpretation is not falsifiably supported. The qualitative example (Fig. 5) is a single case — it shows the mechanism on one instance, not across the margins.\n\nBottom line. This deserves a serious referee. The condition is narrow and fixable: release the blocklist, audit residual leakage (e.g., post-filter retrieval hit-rates on sections mentioning benchmark names), and ideally add an ablation restricting the corpus to confirmed non-matching sections. If that survives, the central claim holds. I would accept conditional on exactly that.","headline":"Solid method advance with unusually careful experiments; the one real gap is the unreleased, unaudited contamination blocklist, which currently keeps the headline claim from being falsifiably separated from memorized benchmark content.","tokens_in":20512,"tokens_out":3518,"duration_ms":20564},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper argues that agent skills improve more when built from retrieved-and-adapted external procedural knowledge than from rollout-only refinement, with gains at both initialization and update.","keywords":["agent skills","skill optimization","retrieval-augmented generation","cross-harness adaptation","LLM agents","skill corpus","procedural knowledge","BM25 retrieval"],"falsifier":"Re-run the four benchmark evaluations after auditing every retrieved section for benchmark-derived text (exact spreadsheet forum prompts, Treasury Bulletin passages, or ALFWorld/WebShop task strings); if removing every section that overlaps such text erases the RASI/RASU advantage over retrieval-free baselines, the result is explained by leakage rather than by transferable skill knowledge.","tokens_in":20996,"feed_emoji":"⚙️","tokens_out":4635,"duration_ms":29246,"temperature":0.7,"pith_summary":"The paper's central claim is that agent skill optimization should draw on an external corpus of shared skills rather than relying only on the agent's own rollouts. It proposes RASO, whose retrieval-augmented initialization (RASI) builds a usable initial skill from task and harness descriptions without any environment rollouts, and whose retrieval-augmented update (RASU) retrieves external knowledge to patch observed failures during iterative refinement. The load-bearing mechanism is Cross-Harness Adaptation, which rewrites skills written for other domains and harnesses into the target harness's vocabulary because on SpreadsheetBench 92.9% of retrieved skills come from a mismatched harness. On four benchmarks and two models, RASO beats retrieval-free optimizers such as TextGrad, GEPA, SkillOpt, and WikiSkill while using equal or fewer rollouts, and the two stages contribute complementary gains. A reader should care because the result points to a cheaper skill-optimization recipe: mine the millions of already-shared skills instead of paying for rollout search.","feed_headline":"Adapted retrieved skills beat rollout-only agent optimizers","feed_subtitle":"RASO grounds skill init and update in an external corpus, beating TextGrad, GEPA, SkillOpt, and WikiSkill across four benchmarks.","key_machinery":"Cross-Harness Adaptation is the shared operation that carries the argument. Given a requirement (a task procedure in RASI, a textual gradient in RASU), the target task description, the target harness description, and the top-K retrieved skill sections, an adaptation agent produces a \"lesson\": a concise procedure re-expressed in the objects, commands, and units of the target harness. It applies three rules — strip domain-specific nouns and procedures without a counterpart in the target harness, focus exclusively on the requirement, and keep tool- or parameter-specific claims only when corroborated by the harness description — which is what converts retrieval from a mismatched source into usable guidance rather than contaminating the skill with inapplicable instructions.","core_discovery":"The central claim is that grounding both skill initialization and skill update in retrieved-and-adapted external knowledge yields higher agent task scores than optimizing solely from execution experience. On the paper's own terms, RASO \"consistently outperforms baselines without retrieval-augmented skill initialization and updating,\" improving over the strongest competing method by up to +6.31 points on SpreadsheetBench with GPT-5.6-Luna and +11.30 on WebShop with Qwen-3.5-9B, while using exactly as many rollouts as TextGrad and WikiSkill and fewer than GEPA and SkillOpt. The paper further shows the gains are additive: on OfficeQA and SpreadsheetBench, RASI alone adds +5.23 and +6.78 points, RASU alone adds +6.86 and +9.40, while the combination reaches +8.33 and +11.66 over the fully retrieval-free baseline, and Cross-Harness Adaptation accounts for most of the RASI gain (+3.88 and +7.74). A qualitative trace attributes one gain to a retrieved satellite-imagery skill whose \"skip empty groups\" rule, once adapted, prevents an index shift in a spreadsheet task.","pith_inferences":["If leakage is controlled, the result suggests a general recipe: any text corpus of procedural documents could replace GitSkills, so manuals, documentation, and web knowledge could be folded into skill optimization without additional rollouts.","Because RASI needs no rollouts, its main cost is the retrieval and adaptation calls themselves; the paper's cost table hints that retrieval-plus-adaptation can beat rollout search on price, but the full crossover point where a cheaper retrieval call should replace a rollout is left implicit.","The K=5 sweet spot for retrieved sections (gains up to 5, mild degradation at 10) suggests a stronger retriever than BM25, or a reranker, could extend the gains; the paper does not test that.","The qualitative example implies the value is in procedure-level structure (e.g., empty-group handling) rather than surface phrasing, which, if true, means the method should transfer to procedural documents that never mention an agent at all."],"forward_implications":["RASI produces a strong initial skill with zero rollouts, beating retrieval-free initialization and direct reuse (SkillRouter) on every benchmark and both backbones, so the first rollout budget can be spent on update rather than initialization.","The two stages are complementary: each independently improves on the fully retrieval-free baseline and the combination is best (+8.33 on OfficeQA, +11.66 on SpreadsheetBench), so neither stage subsumes the other.","RASU alone beats the strongest retrieval-free updater under the same RASI initialization (+1.94 over WikiSkill on OfficeQA, +3.09 over SkillOpt on SpreadsheetBench), showing the update-stage retrieval, not merely a better starting point, drives part of the gain.","Under matched budgets RASO is the cheapest method on three of four benchmarks and uses no more rollouts than the cheapest baselines, so the advantage is not bought with extra environment interaction.","Retrieval helps even from a tiny slice of the corpus: enabling retrieval from just 1% of GitSkills already improves on no-retrieval, and performance grows with corpus size."],"supporting_citations":[{"why":"GitSkills: supplies the external skill corpus that both RASI and RASU retrieve from; the whole framework depends on this corpus existing.","marker":"(Destefanis et al., 2026)"},{"why":"BM25: the retrieval function that selects top-K sections, the retrieval backbone shared by both stages.","marker":"(Robertson & Zaragoza, 2009)"},{"why":"TextGrad: a primary baseline the method must beat and the source of the textual-gradient update style RASU adapts.","marker":"(Yuksekgonul et al., 2025)"},{"why":"GEPA: a baseline that contrastively evolves skills from execution traces without external knowledge.","marker":"(Agrawal et al., 2026)"},{"why":"SkillOpt: a baseline, and the source of the chunked-rollout scheduling and train/validation split convention the paper follows.","marker":"(Yang et al., 2026)"},{"why":"WikiSkill: a baseline whose persistent-wiki update mechanism is the closest retrieval-free analogue to RASU.","marker":"(Tang et al., 2026)"},{"why":"SkillRouter: the direct-reuse baseline used to show that retrieving skills without Cross-Harness Adaptation can hurt performance.","marker":"(Zheng et al., 2026)"}],"fun_headline_variants":["Retrieval-augmented skills beat pure rollout optimization","Mine public skills to kickstart agent optimization","Cross-harness adaptation boosts skill learning without extra rollouts","RASO: retrieved skills outperform rollout-only tuning","Skill initialization from a corpus beats experience-only agents"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The result stands or falls on the per-benchmark blocklist that supposedly strips benchmark-specific skills from GitSkills before retrieval; because the blocklist is not released and residual leakage is not measured, the gains could come from retrieving target-solution content rather than from transferable cross-harness procedural knowledge.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-augmented skills beat pure rollout optimization","Mine public skills to kickstart agent optimization","Cross-harness adaptation boosts skill learning without extra rollouts","RASO: retrieved skills outperform rollout-only tuning","Skill initialization from a corpus beats experience-only agents"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.00016,"raw_usage":{"cost":0.000159628,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000159628,"upstream_inference_prompt_cost":0.000115108,"upstream_inference_completions_cost":4.452e-05},"total_tokens":1186,"prompt_tokens":1027,"completion_tokens":159,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":80}},"tokens_in":1027,"tokens_out":159,"cost_source":"openrouter_usage","duration_ms":2327,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":80,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-30T06:54:43.844522+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-run the four benchmark evaluations after auditing every retrieved section for benchmark-derived text (exact spreadsheet forum prompts, Treasury Bulletin passages, or ALFWorld/WebShop task strings); if removing every section that overlaps such text erases the RASI/RASU advantage over retrieval-free baselines, the result is explained by leakage rather than by transferable skill knowledge.","supporting_citations":[{"cited_title":"OfficeQA.OfficeQA (Opsahl-Ong et al.,","cited_arxiv_id":null,"evidence_quote":"SkillOpt: a baseline, and the source of the chunked-rollout scheduling and train/validation split convention the paper follows."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}