{"arxiv_id":"2609.00549","title":"Skill Following: Evaluating Actual Skill Use in Retrieval-Enabled LLM Agents","abstract":"Large Language Model (LLM) agents increasingly rely on external skills, yet standard evaluations obscure whether retrieving these skills actually helps. Aggregate metrics often compare retrieved versus non-retrieved tasks, introducing severe selection bias and failing to isolate the true effect of skill use. To measure this actual-use capability-which we formalize as Skill Following (SF)-we introduce the Retrieval-Invoked Actual-Use Effect (RAE). RAE computes the same-task outcome difference between matched skill-enabled and skill-disabled executions, conditioned exclusively on tasks where the agent actively retrieved a skill. Evaluating 17 LLMs across coding and mathematical domains, we uncover a stark evaluation paradox: models frequently show positive aggregate retrieval lift but negative RAE. On MBPP+, multiple models that appear to benefit system-wide actually harm their own performance on the exact tasks where retrieval occurred. These findings demonstrate that aggregate averages can create a misleading illusion of tool-use proficiency, whereas RAE directly measures whether the retrieval-to-answer pipeline genuinely rescues more outcomes than it harms.","primary_cat":"cs.CL","cross_cats":[],"submitted_at":"2026-09-01T01:33:57+00:00","latest_version":1,"source":"arxiv","source_label":"arXiv","source_url":null,"has_pdf":true,"has_source":false,"verdict":"ACCEPT","confidence":"HIGH","one_line_summary":"The Retrieval-Invoked Actual-Use Effect (RAE) measures the same-task outcome difference between skill-enabled and skill-disabled executions, and reveals that aggregate retrieval lift frequently masks negative actual skill use across 17 LLMs.","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","upvotes":0,"downvotes":0,"pith_score":0.0,"inbound_count":1,"_links":{"html":"https://pith.science/paper/2609.00549","verdict":"/api/v1/verdicts/2609.00549"}}