{"id":"3595051a-7417-4fc4-a8f4-7580fd6e1c51","arxiv_id":"2611.25429","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":2,"one_line_summary":"Coverage-guided skill self-evolution, using instruction-level labels to target revisions, solves 64.7-70.5% of tasks across three benchmarks, outperforming prior methods by 8.7-14.0 percentage points.","lead":"SkillPlumber lets an agent improve its own natural-language skill instructions by tracking which instructions it actually followed during execution. This coverage-guided approach outperforms existing self-evolution methods across three benchmarks and three different LLM executors.","discovery_kind":"new_method","skeptic_critique":{"audit":{"quote":"The original count therefore resembles a pass@$k$ score, so a skill that passes once by luck still counts as a recovery. To separate reliable repairs from lucky ones, we freeze the skill each method returns on these 52 tasks, execute it three more times with fresh executors, and replay $S_0$ the same way.","status":"unresolved_evidence","question":"Would replaying returned skills on SkillLearnBench and SWE-Skills-Bench-Hard with GPT-5.5 and on all benchmarks with Opus 4.8 and DeepSeek-V4-Pro show persistence similar to the 86.1% observed on SkillsBench with GPT-5.5?","consequence":null,"objection_id":"O1","published_as":"question"},"model":"deepseek/deepseek-v4-flash","headline":"Not confirmed. Pith could not check this concern against the text of the paper it was given; it is listed as an open question, not a finding.","rationale":"","tokens_in":26128,"tokens_out":2439,"duration_ms":37437,"concrete_test":"","verdict_should_be":null,"load_bearing_attack":"Would replaying returned skills on SkillLearnBench and SWE-Skills-Bench-Hard with GPT-5.5 and on all benchmarks with Opus 4.8 and DeepSeek-V4-Pro show persistence similar to the 86.1% observed on SkillsBench with GPT-5.5?","agreement_with_reader":null},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper proposes SkillPlumber, a framework for self-evolving agent skills that uses instruction-level coverage (follow, not-follow, not covered) to guide where skills should be revised. A persistent root cause analysis (RCA) agent turns coverage labels into revision questions, tests revisions with fresh executions, and a maintainer shares investigative strategies across tasks. On three benchmarks (SkillsBench, SkillLearnBench, SWE-Skills-Bench-Hard) with three executors (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro), SkillPlumber solves 134–146 out of 207 tasks, outperforming baselines SkillRevise and CoEvoSkills by 8.7–14.0 percentage points. Ablations show that removing coverage reduces recovered failures by 20.8% on SkillsBench with GPT-5.5.","tokens_in":26428,"tokens_out":2590,"duration_ms":29867,"significance":"If the results hold, this work significantly advances skill self-evolution by introducing a principled signal (instruction-level coverage) that separates whether a failure is due to a wrong instruction or a non-followed one. The experimental design is thorough: three executors, controlled baselines sharing the same initial skill, replay tests, cross-model transfer, and statistical tests (bootstrap, McNemar). The paper identifies a clear limitation of prior methods (whole-run feedback) and addresses it with a concept borrowed from software testing. The findings are likely to influence future work on agent debugging, skill libraries, and self-improving systems.","major_comments":[],"minor_comments":[{"quote":"labels each instruction of a run as follow, not-follow, or not covered, which separates these causes.","comment":"The labels \"follow\", \"not-follow\", and \"not covered\" are used throughout. Consider clarifying early on that these are past-tense descriptions (e.g., \"followed\", \"not followed\") for readability, especially for readers unfamiliar with the prior work on Skill Coverage.","section":"Abstract / Section 1","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"$\\operatorname{Pass}(y_t)$ holds when this feedback reports that $V(x,\\tau_t)$ is a pass.","comment":"The algorithm uses 'Pass(y_t)' as the loop condition. It would be helpful to state explicitly that this checks the verifier output, since earlier the text defines y_t as verifier feedback. The current text in §4.1 says 'Pass(y_t) holds when this feedback reports that V(x,τ_t) is a pass', which is clear but appears after the algorithm.","section":"Section 3, Algorithm 1","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Tasks solved with at most five revisions, and the change over no skill in percentage points.","comment":"The table reports 'Succ.' as raw fractions. For clarity, consider adding a column with the percentage or the percentage-point gain relative to 'No skill' in the table body rather than only in the Δpp column.","section":"Section 5.1, Table 1","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null},{"quote":"Initial failures repaired by each revision round with GPT-5.5 and Codex. A round of CoEvoSkills is one check against the benchmark verifier.","comment":"The caption of the right-side subfigure uses 'Table~\\ref{tab:revisions}' which is fine, but the figure itself is not visible in the supplied text. The caption should state the full meaning of the x-axis (revision round number) for completeness.","section":"Section 5.1, Figure 2 (caption)","consequence":null,"audit_status":null,"objection_id":null,"quote_matched":true,"required_revision":null,"authors_acknowledge":null,"authors_acknowledge_p":null}],"open_questions":null,"recommendation":"accept"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"This paper does something straightforward that turns out to be effective: it takes instruction-level coverage (already used for testing skills) and makes it the guide for skill self-evolution. Instead of judging a whole run as pass or fail, SkillPlumber labels each instruction as follow, not-follow, or not covered, then asks a persistent root-cause-analysis agent to revise where the labels point. A cross-task maintainer shares strategies across tasks. The result: on 207 tasks across SkillsBench, SkillLearnBench, and SWE-Skills-Bench-Hard, SkillPlumber solves the most tasks with all three executors (GPT-5.5, Opus 4.8, DeepSeek-V4-Pro). The gains are consistent; removing coverage costs 20.8% of recovered failures. The experiments are well controlled: same initial skills, same protocol, ablation with a trajectory-analysis control, replay tests, statistical tests. The paper is clearly written and honest about limitations (judge accuracy, token cost, benchmark version differences). The central claim—that coverage-guided revision outperforms outcome-only revision—holds up well.\n\nThe main soft spot is that the coverage labels come from an LLM judge, which can mislabel. The authors acknowledge this and cite validation from prior work, but mislabeling could steer revision in wrong directions. The effect is bounded by the persistent RCA agent that can challenge labels, but the paper does not measure how often mislabeling occurs or whether it harms specific cases. The maintainer contributes only one extra recovery, so the strategy library is not a major driver. The setting (same task for revision and evaluation) follows prior work and is appropriate for reused skills, but it means the results do not test generalization to unseen tasks.\n\nThis paper is for researchers working on LLM agent self-improvement, skill libraries, or test-time adaptation. It deserves a serious referee. The evidence is solid, the method is practical, and the framing of coverage as a revision signal is a genuine step over outcome-only approaches. Recommend peer review.","headline":"SkillPlumber turns instruction-level coverage—a label for how an agent uses each instruction—into a revision signal and shows, across three benchmarks and three executors, that it recovers more failures than existing skill self-evolution methods.","tokens_in":54157,"tokens_out":1373,"duration_ms":21818},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The authors report that SkillPlumber uses instruction-level coverage to guide self-evolution of agent skills, outperforming existing methods on three benchmarks with three executors.","keywords":["agent skills","self-evolution","instruction-level coverage","root cause analysis","LLM agents","skill revision","coverage-guided debugging"],"falsifier":"A human evaluation of the coverage judge on a sample of SkillsBench tasks, showing that the judge frequently mislabels instructions (e.g., labeling a followed instruction as not-follow), would contradict the method's assumption that coverage provides a reliable signal. Alternatively, an experiment where coverage labels are replaced with random labels of the same distribution and the method still matches SkillPlumber's recovery would suggest the gain is not due to the label content.","tokens_in":27940,"feed_emoji":"🤖","tokens_out":1832,"duration_ms":26902,"temperature":0.2,"pith_summary":"Agent skills are natural-language instructions that an agent follows during a task. A skill helps only when its instructions are correct and properly followed. Existing self-evolution methods judge an entire run as pass or fail, but cannot tell whether a failure came from a wrong instruction or from the agent ignoring a correct one. SkillPlumber leverages instruction-level coverage, which labels each instruction as follow, not-follow, or not covered, to separate these causes. A persistent root cause analysis agent turns each label into a revision question and checks revisions against subsequent runs; a maintainer distills strategies across tasks. The authors claim that SkillPlumber solves 64.7% to 70.5% of tasks across three benchmarks and three executors, 25.6 to 30.4 percentage points above the initial skill and 8.7 to 14.0 percentage points above the strongest baseline, and that removing coverage reduces recovered failures by 20.8% on SkillsBench with GPT-5.5.","feed_headline":"Coverage-guided skill evolution beats whole-run methods","feed_subtitle":"SkillPlumber uses instruction-level labels to decide which part of a skill to revise, recovering more failed tasks with fewer revisions.","key_machinery":"Instruction-level coverage (Skill Coverage) labels each instruction of a skill as follow (the condition occurred and the agent acted as required), not-follow (the condition occurred but the agent did not comply), or not covered (the condition never arose). SkillPlumber turns these labels into revision questions: for a not-follow instruction, ask why it was not followed; for a followed instruction, ask whether it is itself wrong; for a not covered instruction, assess whether it is relevant. A persistent root cause analysis agent reviews its own previous hypotheses and predictions against the next run, and a maintainer accumulates investigative strategies across tasks.","core_discovery":"The paper claims that instruction-level coverage, which labels each skill instruction as follow, not-follow, or not covered from an execution trace, can be used as a signal for skill self-evolution. By directing revision toward the instructions the agent did not follow, rather than judging the run as a whole, SkillPlumber recovers more failed tasks than methods that rely on outcome-only or trajectory-level diagnosis. The paper further claims that the coverage signal, not the agentic revision budget or the strategy library, drives the gain: removing coverage reduces recoveries, while removing the maintainer has little effect.","headline_check":{"rule":"report_a_claim_v1","sent_back":false,"unattributed":{}},"pith_inferences":["The approach treats skill revision as analogous to program debugging with code coverage; this analogy may inspire further work on test adequacy metrics for agent behaviour, such as branch coverage or mutation testing for skill instructions.","Because the coverage judge is the same model as the executor, mislabelling could be systematic; future work could explore using a separate, specialised judge or incorporating human-in-the-loop verification for critical instructions.","The cross-model transfer experiment suggests that coverage-driven revisions capture some executor-specific behaviour; reusing skills across different models may require re-running the revision loop for each target model, not just transferring final skills."],"forward_implications":["If SkillPlumber’s coverage-guided revision is correct, then self-evolving agents can distinguish between instruction errors and execution errors, leading to more reliable skill improvement without human intervention.","The method provides a way to use execution traces from deployment (where no verifier exists) to improve skills, since coverage requires only the trace, not the outcome.","The persistent RCA agent can retract its own earlier revisions when they do not work, which may reduce the number of wasted revision rounds compared to methods that always spend the full budget.","Combining instruction-level coverage with code coverage of attached scripts could allow revision of the entire skill bundle, not just natural-language instructions."],"supporting_citations":[{"why":"Introduces instruction-level coverage for agent skills, which SkillPlumber uses as its main signal for revision.","marker":"tan2026skillcoverage"},{"why":"Baseline method that revises skills guided by repair principles and trace analysis; its authoring stage generates the initial skills for all methods.","marker":"liu2026skillrevise"},{"why":"Baseline method that co-evolves a surrogate verifier alongside the skill; used as a comparison in the evaluation.","marker":"zhang2026coevoskills"},{"why":"One of the three evaluation benchmarks, measuring how well agent skills work across diverse tasks.","marker":"li2026skillsbench"},{"why":"Second evaluation benchmark, for continual learning of agent skills on real-world tasks.","marker":"zhong2026skilllearnbench"},{"why":"Third evaluation benchmark, for skills in real-world software engineering; the hard subset is used.","marker":"han2026sweskillsbench"},{"why":"Inspiration for using coverage in fault localisation, analogous to using instruction coverage for skill revision.","marker":"jones2005tarantula"}],"cache_read_input_tokens":0,"weakest_assumption_plain":"The coverage labels (follow, not-follow, not covered) come from an LLM judge that may mislabel an instruction, and if the judge consistently mislabels, the root cause analysis agent will pursue wrong revision directions, potentially harming performance or wasting the revision budget.","cache_creation_input_tokens":0},"created_at":"2026-11-20T14:20:38.871903+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium+deepseek/deepseek-v4-flash+p.d60f54226f00","input_kind":"tex","input_chars":79321,"input_total_chars":79321,"passes_missing":[],"published_standing":"established","published_confidence":"moderate","objection_audit":{"version":"pith-objection-audit@1","counts":{"raised":1,"confirmed":0,"questions":1,"withdrawn":0,"moved_to_minor":0,"math_error_not_certified":0,"withdrawn_reviewer_error":0,"typesetting_minor_dropped":0,"withdrawn_already_addressed":0},"open_questions":[{"origin":"skeptic","reason":"not_settled","section":null,"question":"Would replaying returned skills on SkillLearnBench and SWE-Skills-Bench-Hard with GPT-5.5 and on all benchmarks with Opus 4.8 and DeepSeek-V4-Pro show persistence similar to the 86.1% observed on SkillsBench with GPT-5.5?","objection_id":"O1"}],"standing":{"basis":["first_pass_verdict"],"standing":"established","confidence":"moderate"},"standing_rule":"1. No standing is published (not_assessed) when the review read only the abstract, when the referee pass or the objection check did not run, or when the first pass gave no verdict. 2. The first pass's verdict sets the start: ACCEPT is established, CONDITIONAL is established_narrower, REJECT is not_supported. 3. Each objection that survived the check against the paper's text caps the standing: one that breaks the central claim caps it at not_supported, one that narrows it at established_narrower, one that weakens it at supported_with_gaps. The standing is the lowest of the start and the caps. 4. A first-pass REJECT stands only if a surviving objection breaks the central claim. Without one, the standing is the lowest cap of the surviving objections, or not_assessed when none survived. 5. The confidence is the lowest of the first pass's confidence and the confidence of each objection that set the standing, and is no higher than moderate while a question is open or part of the paper's text was not supplied. 6. If any objection was withdrawn because Pith's reviewer misread the paper, the confidence is no higher than moderate, whatever the first pass said: a review that misread the paper once may have misread it elsewhere.","input_kind":"tex","nothing_to_audit":false,"bar":{"enabled":false}},"run_record":{"models":["deepseek/deepseek-v4-flash"],"passes":{"reader":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.001669,"tokens_in":30942,"tokens_out":3802,"duration_ms":53753,"input_sha256":"b84a5e7cda19dc29a72173c19fff373c9a12fd36d70209e9effc2b661590b22c","prompt_sha256":"f9ead730038216984cfb56d7cb34332bdf691828296a1de4245f3d6667d4bf56","message_sha256":"a9f95dc1e6de60f48590bf0719107c9d3acb6caeaa90fd800d644a880c82c6b9","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"referee":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.001457,"tokens_in":26428,"tokens_out":2590,"duration_ms":29867,"input_sha256":"b84a5e7cda19dc29a72173c19fff373c9a12fd36d70209e9effc2b661590b22c","prompt_sha256":"a852d99aea0ddfd900e1483f76e20777aae3afde9c016d5b8d6e5db01064c728","message_sha256":"931ac706f37a06f3c15464550adf62af444ce5cf6ba925ed072ff7703d4b4064","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"skeptic":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.001467,"tokens_in":26128,"tokens_out":2439,"duration_ms":37437,"input_sha256":"b84a5e7cda19dc29a72173c19fff373c9a12fd36d70209e9effc2b661590b22c","prompt_sha256":"8f88c0f61488b1e764c317998cddf35fffc33a1f20a3b1be54330d7843386fc7","message_sha256":"58708fd4cef96c2b4ac40d91b3e5d8657b8021b0c82874b462b4ccd9a23bd7c9","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"circularity":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.001336,"tokens_in":26307,"tokens_out":1373,"duration_ms":16424,"input_sha256":"b84a5e7cda19dc29a72173c19fff373c9a12fd36d70209e9effc2b661590b22c","prompt_sha256":"e9946e39d67b76cc60476379bc32abdc44665b5f0c438659796547bb1041a01f","message_sha256":"550e35af023124369b66b917ba83d3d7aa8e6007176dab9a1e0e6bb58854daa8","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"desk_editor":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.001762,"tokens_in":54157,"tokens_out":1373,"duration_ms":21818,"input_sha256":"b84a5e7cda19dc29a72173c19fff373c9a12fd36d70209e9effc2b661590b22c","prompt_sha256":"e7211f6a3d2734c1025ce7e7e6d48f78b0944d0a32d5021f81be63b27721420a","message_sha256":"c281666ebb824a8ecbff3e6ee81e805e146b67ce716b401bf1cd93f680b4ab99","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"objection_audit":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.001594,"tokens_in":26767,"tokens_out":2558,"duration_ms":37221,"input_sha256":"b84a5e7cda19dc29a72173c19fff373c9a12fd36d70209e9effc2b661590b22c","prompt_sha256":"6c7a6883d5517cf9e4945d83947200a36dcd152b8b56c32baf7500d8b3841d0a","message_sha256":"f1f3767d3d5add7f21ce1746ec5628c1f5ee8798baffea1455f22e8b4eedd476","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"pith_extraction":{"model":"deepseek/deepseek-v4-flash","effort":"medium","cost_usd":0.001496,"tokens_in":27940,"tokens_out":1832,"duration_ms":26902,"temperature":0.2,"input_sha256":"b84a5e7cda19dc29a72173c19fff373c9a12fd36d70209e9effc2b661590b22c","prompt_sha256":"e4f56bf568e7854d0a1d2efb6ec96f8697d5132737cdb31cb7fc64acbd923974","message_sha256":"e4075c63ab565e0cf610c37507aad7133a702d408d7f035d9e08c418911ccd36","cache_read_input_tokens":0,"cache_creation_input_tokens":0}},"version":"pith-run-record/v1","prompt_set_sha256":"d60f54226f003b7308ab6f0f9f2951c09c726f00cc66fc05af3456f212247e93","configured_pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium"},"basis":{"kind":"full_text","source":"recorded","chars_given":79321,"chars_total":79321},"receipt":{"review_sha256":"3236d26d10facc25b23cb915eb88c4d553fc682688b85c889e8aa9f8b27cf4e3","rule":"pith-review-sha256/v1","hashed_at":"2026-11-20T14:23:34.857284+00:00","fields":["id","arxiv_id","paper_version","created_at","pipeline_version","model_set","run_record","input_kind","input_chars","input_total_chars","passes_missing","verdict","confidence","novelty_score","correctness_risk","formal_verification","parameter_count","discovery_kind","one_line_summary","lead","skeptic_critique","referee_report","author_rebuttal","desk_editor","pith_extraction","objection_audit","published_standing","published_confidence"],"matches_current":true},"falsifier":"A human evaluation of the coverage judge on a sample of SkillsBench tasks, showing that the judge frequently mislabels instructions (e.g., labeling a followed instruction as not-follow), would contradict the method's assumption that coverage provides a reliable signal. Alternatively, an experiment where coverage labels are replaced with random labels of the same distribution and the method still matches SkillPlumber's recovery would suggest the gain is not due to the label content.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":"tex"},"url":"https://pith.science/accuracy"},"review_version":1}