{"paper":{"title":"Evolve Where It Matters: Coverage-Guided Self-Evolution of Agent Skills","arxiv_id":"2611.25429","primary_cat":"cs.CL","submitted_at":"2026-11-19T03:11:24+00:00","version_compiled":"v1"},"notice":"Every quoted sentence is the authors' text from the arXiv listing abstract. Labels and probabilities are the outputs of classifier-1.13.0 on the question bank abstract-p1-20260923, exactly as returned. No sentence in this record was written by a language model.","schema":"pith.compiled-record/v0","status":"compiled","abstract":{"source":"arXiv listing abstract v1","sentences":[{"i":0,"p":1.0,"id":"s0","own":false,"role":"background","text":"Agent skills package procedural knowledge as natural-language instructions, and a skill helps only when its instructions are correct and properly followed by the agent.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.07,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.0,"result":0.0,"background":1.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":1,"p":0.95,"id":"s1","own":false,"role":"gap","text":"Self-evolving agents revise their own skills from the runs they make, but existing methods judge each run as a whole, by its outcome or by where it went wrong, and cannot tell whether a failure comes from a wrong instruction or from one the agent did not follow.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.07,"main_claim_p":0.03,"quantitative":false,"quantitative_p":0.08,"role_probabilities":{"gap":0.95,"method":0.0,"result":0.0,"background":0.05,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}},{"i":2,"p":0.89,"id":"s2","own":false,"role":"background","text":"Instruction-level coverage was introduced to test skills and labels each instruction of a run as follow, not-follow, or not covered, which separates these causes.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.09,"main_claim_p":0.04,"quantitative":false,"quantitative_p":0.06,"role_probabilities":{"gap":0.01,"method":0.09,"result":0.0,"background":0.89,"limitation":0.0,"future_work":0.0,"contribution":0.01,"interpretation":0.0}},{"i":3,"p":0.85,"id":"s3","own":true,"role":"contribution","text":"We leverage it for skill self-evolution with SkillPlumber, a framework in which an agent evolves its own skills where coverage shows they fall short.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.9,"main_claim_p":0.11,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.15,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.85,"interpretation":0.0}},{"i":4,"p":0.98,"id":"s4","own":true,"role":"method","text":"A persistent root cause analysis agent turns each label into a revision question and checks each revision against the next run, while a maintainer distills investigations across tasks into a shared strategy library.","hedge":{"p":0.99,"label":"asserted"},"own_p":0.83,"main_claim_p":0.05,"quantitative":false,"quantitative_p":0.03,"role_probabilities":{"gap":0.0,"method":0.98,"result":0.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.02,"interpretation":0.0}},{"i":5,"p":1.0,"id":"s5","own":true,"role":"result","text":"Across three benchmarks and three executors, SkillPlumber solves 64.7% to 70.5% of the tasks, 25.6 to 30.4 percentage points higher than the initial skill and 8.7 to 14.0 percentage points higher than the strongest baseline, and removing coverage reduces recovered failures by 20.8% on SkillsBench with GPT-5.5.","hedge":{"p":1.0,"label":"asserted"},"own_p":0.96,"main_claim_p":0.74,"quantitative":true,"quantitative_p":0.99,"role_probabilities":{"gap":0.0,"method":0.0,"result":1.0,"background":0.0,"limitation":0.0,"future_work":0.0,"contribution":0.0,"interpretation":0.0}}]},"compiled":{"main_claim":{"p":1.0,"text":"Across three benchmarks and three executors, SkillPlumber solves 64.7% to 70.5% of the tasks, 25.6 to 30.4 percentage points higher than the initial skill and 8.7 to 14.0 percentage points higher than the strongest baseline, and removing coverage reduces recovered failures by 20.8% on SkillsBench with GPT-5.5.","location":"abstract s5","hedge_strength":{"p":1.0,"label":"asserted"}},"provenance":{"route":{"via":"TypeSafe API","model":"classifier-1.13.0","endpoint":"https://api.typesafe.ai/v1/systemone","provider":"TypeSafe"},"packing":"one request per paper","requests":1,"input_tokens":5195,"cost_usd_reported":0.0},"compiled_at":"2026-11-20","bank_version":"abstract-p1-20260923","model_version":"classifier-1.13.0","abstract_states":{"claims_first":0.08,"states_limitation":0.23,"human_participants":0.04,"controlled_comparison":0.95,"randomized_assignment":0.02,"states_quantitative_result":0.99,"compares_with_prior_results":0.98,"mentions_code_or_data_availability":0.04},"evidential_basis":{"p":0.97,"value":"observation or experiment","probabilities":{"argument":0.0,"literature":0.0,"proof_or_derivation":0.0,"analysis_of_existing_data":0.0,"observation_or_experiment":0.97,"simulation_or_computation":0.03}},"contribution_type":{"p":1.0,"value":"method or tool","probabilities":{"review":0.0,"method_or_tool":1.0,"empirical_study":0.0,"simulation_study":0.0,"theoretical_result":0.0,"dataset_or_benchmark":0.0,"position_or_commentary":0.0}},"contribution_statement":{"p":0.98,"text":"We leverage it for skill self-evolution with SkillPlumber, a framework in which an agent evolves its own skills where coverage shows they fall short.","location":"abstract s3"}},"coverage":{"tier":"abstract","levels_run":["P abstract"],"not_examined":["full text","figures and tables","references and citing sentences","relations to other papers"],"sources_read":["arXiv listing abstract v1"],"levels_not_run":["S sections","T body sentences","C citations","R cross-paper relations"],"sentences_examined":6,"sentences_in_abstract":6},"measured_accuracy":{"url":"https://pith.science/accuracy","measured_on":"2026-10-02","method":"two independent model families labelled a fresh random sample blind; the figure is how often the published label matches their consensus","sentence_role_agreement":0.92,"main_claim_agreement":0.908,"typed_citation_agreement":0.933,"typed_citation_agreement_95_interval":[0.917,0.945],"review_comments_accurate":0.758,"review_comments_accurate_95_interval":[0.684,0.82],"review_caught_admitted_error_found_or_related":0.448,"review_comments_judged":{"judged_by":["gpt-5.6-luna","grok-4.7"],"judged_against":"Pith's own text copy of each paper: text extracted from the PDF, the first 90,000 characters","measured_on":"2026-10-04","accurate":113,"decided":149,"sampled_reviews":200,"split_by_lane":false,"by_lane":"https://pith.science/accuracy#by-lane"}}}