Artificial Intelligence · arXiv:2610.32322 v1 · submitted 23 Oct 2026 · 41 references
Beyond Gain: Measuring Skill Internalization for Agent Skill Retirement
The paper says
“LLM agents retrieve skills from public hubs that hold up to about 100,000 entries and serve many models.”
more of the abstract
As models improve, a hub must find the skills a given model has internalized, meaning it can do without them, and retire them for that model, yet internalization has not been measured with known reliability. Published rules that compare runs with and without a skill retire on the gain alone. The gain is near zero both for an internalized skill and when neither the model nor the skill delivers the capability, a state we call uncovered. We instead withhold the skill, score the model on a task synthesized from it, and trust a high unaided score only when another model gains from the skill, since nearly half of high scores come from tasks no model benefits from. Under four models, a published gain rule applied with one task per skill removes over half of the uncovered skills. On SkillsBench, 28 of its 31 retirements rest on probes no run passed in full, while retiring on the unaided score costs no detectable reward. Unaided verdicts reproduce across tasks more often than the gain rule's (Cohen's kappa 0.42 against 0.23) but differ across models, so a hub must read them per model. We release 1,251 skills with tasks and item scores under four models.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 26 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. The blind-spot diagnosis is sound and the measurement practice is genuinely careful, but the paper's own reliability numbers put the validity gate — its load-bearing piece — on shaky ground. This line is wrong
3 major objections · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 41 references say.
The paper says, at §2.4, Eq. (2), with §4.1
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The validity gate is the instrument that separates 'the model has internalized the capability' from 'the task does not require the skill,' but its admission decision is a threshold on the witness gain measured on one synthesized task per skill.
more of objection 1
The paper's own parallel-forms reliability for the gain in Figure 3 is only 0.15, and the reliability of the gate decision itself is never reported. Because the gate admits only 52.6% of high-U tasks, admitting or rejecting a task is a high-stakes binary decision made on a measurement whose reproducibility is the weakest reported quantity in the paper. The released two-task data (124 skills scored on both tasks) should allow the authors to report how often the gate's admission decision flips when the task is swapped, and to give a kappa for gate admission; without this, the gate's behavior may be a one-task artifact. The paper's note that a single witness can be insufficient (Appendix B) addresses witness coverage, not the reliability of the gain threshold in Eq. (2).
The paper says, at §2.4, with §4.4
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
Even if the witness gain were perfectly reliable on a fixed task, a witness gain is model-relative. Section 4.4 reports that verdicts flip on 14–23% of skills across model pairs and that 19–32% of skills retired for one model are skills that work for another (A >= 0.7, U < 0.4).
more of objection 2
A witness model's gain therefore does not establish that the admitted task requires the skill for the target model; the target may pass unaided because the task is easy for it, not because the capability is internalized. The manuscript acknowledges this heterogeneity, but it does not offer a target-specific validation of the gate. A concrete and available test is to use the 124 two-task skills to check whether gate-admitted high-U skills retain high U and the retire verdict on a second task, and to repeat this for each model as target. The text-predictor check in Appendix B addresses triviality predictable from text; it does not address target-specific triviality.
The paper says, at §4.1, judge-dependence paragraph
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The judge-dependence analysis reports that unsupported full marks near the threshold lift U by about 0.1, which is the same magnitude as the gate threshold gamma and the tau-minus-h margin. The paper reports aggregate judge agreement and the 20/18 split, but not how many individual near-threshold level-rule verdicts or gate admissions would flip under alternate scoring.
more of objection 3
Since both the level rule and the gate make discrete decisions at exactly this margin, a per-skill sensitivity analysis is needed before the 'certificate' language is fully supported. This is a second, measurement-level threat to the reliability of the gate and to the precision of the level rule near the retirement threshold.
3 minor objections
The paper says, at Eq. (13)
Pith says: minor objection 1 of 3
The four covariance terms are listed without signs. State explicitly that the identity uses 0.044 + 0.046 - 0.043 - 0.036 = 0.011, or provide the covariance matrix, so that the near-cancellation between the two arms is transparent and checkable.
The paper says, at §3.3
Pith says: minor objection 2 of 3
The sentence saying that some audited rules 'disclaim the reading we test' would benefit from naming those rules; Table 2 does not carry the disclaimers, so the reader cannot tell which rows are being qualified.
show 1 more
The paper says, at Figure 3
Pith says: minor objection 3 of 3
The figure reports kappa bars with raw agreement values above them, but the legend does not define 'level' and 'band' verdicts on the figure itself; adding a one-line definition would make the 0.42 vs 0.23 comparison self-contained.
What would settle it
Compute the parallel-forms agreement of the gate decision itself: on the 135 skills that carry two tasks (§4.1), apply Eq. (2), with γ fixed at 0.1 and, for robustness, swept over 0.05–0.2, separately to each task's witness gains, and count how often the admission decision (admit/reject) flips between the two tasks. Restrict to the skills with U≥0.8 on at least one task, since those are the cases where the gate actually governs a retirement decision. If the gate flips on more than roughly 20–25% of those skills, single-task gate admission cannot support a per-model internalization certificate, and the paper should report multi-task admission rates (as in Algorithm 1) before claiming that the level rule retires correctly on gated tasks.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 26 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Short version: the formal diagnosis is right and the empirical work is the most careful this subfield has produced, but the positive protocol leans on a validity gate whose reliability is never measured, and the paper's own numbers suggest why that matters.
What's new: Proposition 1 — any rule that vanishes on the diagonal pA = pU cannot separate an internalized from an uncovered skill — is clean, short, and checkable, and it's not present in the cited curation literature. The three-state framing (internalized / skill works / uncovered), the per-model unaided-label idea, and the level rule are genuinely new protocol. The audit practice deserves real credit: bootstrap intervals, paired comparisons, the cross-fitted audit where verdict and state are read on different tasks, a pre-registered SkillsBench exclusion, an independent judge re-scoring, and a released dataset with 1,251 skills. This paper treats measurement validity as the problem, not an afterthought.
The stress-test concern about the gate lands. The gate admits a task only when some witness gains more than 0.1 from the skill on one synthesized task per skill, yet the paper reports the gain's parallel-forms reliability at 0.15 and that unsupported full marks lift U near tau by about 0.1 — the same magnitude as gamma. The paper acknowledges the witness problem and the per-model flip in verdicts, but it never reports the reliability of the gate decision itself. That is a genuine hole in the positive claim, not a cosmetic one: without the gate, "the level rule certifies internalization" collapses to "the model scores high on this one task." The gate is load-bearing and should carry its own reliability coefficient.
I'd push back on one part of the stress test: the gate weakness doesn't touch the central diagnosis. Proposition 1 stands independently, and the composition evidence (band retires 57.5% of the uncovered state; 28 of 31 SkillsBench band retirements rest on probes no run passed in full) is solid. Two smaller soft spots: the benchmark cost comparison is a null result that can't discriminate between rules — the paper says this itself, but it means "retire on level, costs nothing" rests more on the audit's composition numbers than on downstream execution — and the kappa advantage (0.42 vs 0.23) has a confidence interval that barely excludes zero. The benchmark should be paired with a stated minimum detectable effect.
Who it's for: anyone working on skill curation, library lifecycle, or agent evaluation. It deserves a serious referee. I'd send it to review, with the request that a revision measure gate-decision reliability directly and state the benchmark's minimum detectable effect.
The strongest objection
The paper's positive claim—that the level rule, read on a validity-gated task, certifies internalization—stands or falls with the validity gate of Eq. (2), the only instrument that separates 'the model has the capability' (internalized, retire) from 'the task does not require the skill' (trivial, keep). The gate admits a task when some witness gains more than γ=0.1 on one synthesized task per skill (§2.4). Three facts make this the least secure load-bearing condition. First, the paper itself measures the gain's parallel-forms reliability at 0.15 (§4.1, Fig. 3), so the gate's evidence is nearly unreproducible when the task is swapped; the gate is nevertheless applied with one task per skill in the main analysis. Second, the witness is a different model, and the paper's own §4.4 shows verdicts flip on 14–23% of skills across model pairs and that 19–32% of skills retired for one model work for another, so a witness's gain does not show the task is non-trivial for the target. The paper acknowledges both facts (sentence b350; §4.4) but never measures the reliability of the gate decision itself. Third, γ=0.1 is the same magnitude as the judge's unsupported-full-mark error, which 'lift U near τ by about 0.1' (§4.1), so gate decisions sit within one judge error of flipping. Since the gate rejects 47.4% of high-U tasks but admits 52.6% of them, a noisy gate mis-admits a non-trivial share of the 26.4% of skills the level rule retires, producing false internalization certificates. The cross-fitted audit (level 14% vs band 45%, paired difference −31 [−52,−7]) mitigates the definitional circularity between the states and the rule, but it runs on only 29 decisions and still relies on the gate.
The diagnosis portion of the paper is solid and earns credit: Proposition 1 correctly shows that any decision rule depending only on a criterion vanishing on the diagonal p_A=p_U cannot separate internalized from uncovered skills, and Proposition 2 shows that more runs remove noise but not the error itself. The empirical audit is consistent with this—the gain band retires 57.5% of the uncovered state on capability scores and 80–99% on binary outcomes, while the level rule retires none. The binary collapse mechanism of §3.3 (Eq. 9) is clean and verifiable. Independent support is present: a released dataset and pipeline, bootstrap and paired intervals, an independent judge re-scoring (correlation 0.89, verdict agreement 91.4%), and a second-model-family regrading near the threshold. The load-bearing weakness is not the diagnosis but the paper's own positive instrument: the level rule's certificate of internalization is only as good as the validity gate that rules out the trivial-task explanation for a high unaided score. That gate uses a single-task witness gain whose reproducibility the paper itself reports as nearly zero (cross-task reliability 0.15), whose threshold γ=0.1 matches the magnitude of the judge's measured unsupported-mark error, and which comes from a different model than the target. The paper acknowledges the components individually (single witnesses insufficient in b350; cross-model verdict heterogeneity in §4.4; judge error in §4.1) but never reports the reliability of the gate admission decision itself, which is the quantity that would settle whether a high U on an admitted task reflects internalization or residual task easiness for the target. This is an empirical, fixable gap, not a logical flaw, so the appropriate disposition is conditional acceptance pending that measurement, which is exactly the reader's CONDITIONAL verdict. My concern does not move the verdict; it pins the condition. I agree with the reader's weakest_assumption, which identifies the same gate as the least secure step.
The referee's report
If the validity gate can be shown to be reliable, the paper makes a substantive contribution: it gives a precise, minimal formal argument that gain-only retirement rules are blind to the uncovered state, and it provides an unusually careful empirical audit with bootstrap intervals, paired comparisons, a pre-registered benchmark exclusion, an independent judge check, and a released dataset and codebase. The reliability decomposition and the proof that tasks buy more than runs (Proposition 3) are useful beyond this specific setting. The central caveat is that the paper's positive claim—that a validity-gated high unaided score certifies internalization—depends on the gate in Eq. (2), whose decision reliability is not directly reported. That issue is load-bearing but appears addressable with data the authors already have, so it warrants revision rather than rejection.
Pith's reading of the paper (a machine's summary, not the authors')
Public skill hubs serve many models, and a curator must retire a skill for a model once that model has internalized it — absorbed it so completely that retrieving it adds nothing. The paper's diagnosis is that the standard rule, retire when the aided-minus-unaided gain ($A-U$) is near zero, cannot separate an internalized skill from an uncovered one, in which neither the model nor the skill delivers the capability, because both states sit on the diagonal where the two arms agree. It proves that any criterion vanishing there, including leave-one-out and Shapley attributions on a single-skill task, must give one verdict for the whole diagonal and so risks certifying a capability the model lacks or hiding one it has; more runs remove only noise, not the error. Its remedy is to certify unaided capability directly: withhold the skill, score the model on an executable task synthesized from the skill by another model family, and trust a high score only when a witness model demonstrably gains from the skill on the same items. Across four models the gain-band rule retires more than half of the uncovered skills, while retiring on the unaided level costs no detectable reward and reproduces across tasks more often, so the paper concludes a hub should retire per model on a label of known reliability.
What carries the argument. The load-bearing object is the three-state partition of skills by their aided score $A$ and unaided score $U$ — internalized (both high), skill works ($A$ high, $U$ low), uncovered (both low) — together with the validity gate that makes the unaided level readable. The gate admits a task only when some witness model gains more than $0.1$ from the skill on the same rubric items (Eq. 2), because nearly half of high unaided scores come from tasks no model benefits from. Three propositions carry the argument: Proposition 1 bounds the risk of any criterion vanishing on the diagonal $p_A = p_U$, Proposition 2 shows that replication converges to that same error rather than removing it, and Proposition 3 gives the measurement budget, decomposing label noise into skill, task, and run components (34%, 39%, 27% of single-run variance) and showing that tasks buy more reliability than runs, with $\sigma_e^2 = \sigma_t^2/n_t + \sigma_r^2/(n_t n_r)$ and an optimal allocation of $n_r^* = \sqrt{\sigma_r^2 c_t / (\sigma_t^2 c_r)}$ runs per task. An identity for the gain measured on two tasks, $\mathrm{Cov}(G,G') = \mathrm{Cov}(A,A') + \mathrm{Cov}(U,U') - \mathrm{Cov}(A,U') - \mathrm{Cov}(U,A')$, explains why the gain's cross-task reliability (0.15) is far below the level's (0.41): whatever a skill carries into both arms on every task cancels in the difference.
Core claim. The paper's claim is that skill retirement is a per-model certificate of unaided capability, not a statement about what the skill contributes. With aided score $A$ and unaided score $U$, a skill is internalized when both are high, it works when $A$ is high and $U$ is low, and it is uncovered when both are low; the gain $A-U$ is near zero in the internalized and uncovered states alike. A retirement rule that reads only the gain, including leave-one-out and Shapley attributions on a single-skill task, therefore issues one verdict for the entire diagonal $p_A = p_U$ and necessarily has positive risk either of certifying a capability the model lacks or of hiding one it has (Proposition 1), and replication drives the risk to that same error rather than to zero (Proposition 2). The paper's alternative reads the unaided level on a task that states the goal but not the content the skill supplies, admits the task only when some witness model gains more than $0.1$ from the skill on the same rubric items, and retires at $U \geq 0.8$. It reports that under all four models the band rule $|A-U| < 0.10$ removes 57\textendash65% of the uncovered state, that on SkillsBench 28 of the band's 31 retirements rest on probes where no run of either arm passed in full, and that deleting what the level rule retires leaves benchmark reward unchanged.
The premise it rests on. The load-bearing premise is that a witness model's gain of more than 0.1 on a task proves the task genuinely requires the skill, so the target model's high unaided score is evidence of internalization rather than of an easy task — yet the paper itself reports that gains are far less reproducible across tasks than levels (0.15 against 0.41, Section 4.1) and that witnesses disagree, so a task admitted by the witness set may still be one the target passes for reasons of task difficulty.
If the paper is right (Pith's inferences, not the authors')
A hub should retire a skill for a model on the unaided level with a validity gate, read per model; the gain remains useful for ranking skills but is the wrong quantity for removing them.
The variance decomposition fixes a measurement budget: at observed costs one run per task is optimal, and extra tasks belong near the retirement threshold, since three tasks with one run each misclassify fewer skills than one task run three times.
Band-style gain rules that use an all-items binary pass criterion collapse whenever no run passes every item — 28 of the band's 31 SkillsBench retirements — so those retirements rest on two failed arms, not on evidence of internalization.
A verdict cannot be transferred across models: 37% of skills receive contradictory verdicts among the four models, and the disagreement lies in the skill-by-model interaction rather than in model strength.
The level verdict reproduces across tasks more often than the band's (Cohen's kappa 0.42 against 0.23), so reliability as well as validity favours the level rule.
The paper removes the gain from the decision but not from the pipeline: the validity gate is itself a gain measurement, used to verify that the task needs the skill rather than to certify the model. The implied division of labour — gain detects task–skill dependence, level certifies model capability — is a testable design principle for other pair-comparison evaluations, such as tool selection or r
The diagonal-blindness result generalizes: any attribution that is a pure contrast between two conditions carries the blind spot wherever the two conditions agree for opposite reasons. The same argument should transfer to guardrail comparisons or preference audits that must distinguish "succeeds in both cases" from "fails in both cases".
The uncovered state is the most actionable output of the protocol: those skills are repair candidates, not retirement candidates, and the procedure already flags them, so a hub could act on repair before deciding on retirement.
Because the cross-model disagreement lies in the skill-by-model interaction, no global threshold shift aligns the models; following the paper's own logic, a hub must re-read the label for each new model version, not only for each model family.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
LLM agents retrieve skills from public hubs that hold up to about 100,000 entries and serve many models. As models improve, a hub must find the skills a given model has internalized, meaning it can do without them, and retire them for that model, yet internalization has not been measured with known reliability. Published rules that compare runs with and without a skill retire on the gain alone. The gain is near zero both for an internalized skill and when neither the model nor the skill delivers the capability, a state we call uncovered. We instead withhold the skill, score the model on a task synthesized from it, and trust a high unaided score only when another model gains from the skill, since nearly half of high scores come from tasks no model benefits from. Under four models, a published gain rule applied with one task per skill removes over half of the uncovered skills. On SkillsBench, 28 of its 31 retirements rest on probes no run passed in full, while retiring on the unaided score costs no detectable reward. Unaided verdicts reproduce across tasks more often than the gain rule's (Cohen's kappa 0.42 against 0.23) but differ across models, so a hub must read them per model. We release 1,251 skills with tasks and item scores under four models.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-26
The authors' main claim, as the classifier picked it out 0.80
“We instead withhold the skill, score the model on a task synthesized from it, and trust a high unaided score only when another model gains from the skill, since nearly half of high scores come from tasks no model benefits from.”
How the main claim is stated: asserted 0.97
What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.91 · human participants 0.05 · controlled comparison 0.79 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.84
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| LLM agents retrieve skills from public hubs that hold up to about 100,000 entries and serve many models. | background 1.00 | 0.08 | 0.02 | asserted 0.97 |
| As models improve, a hub must find the skills a given model has internalized, meaning it can do without them, and retire them for that model, yet internalization has not been measured with known reliability. | gap 0.85 | 0.08 | 0.08 | asserted 0.98 |
| Published rules that compare runs with and without a skill retire on the gain alone. | background 0.85 | 0.05 | 0.04 | asserted 1.00 |
| The gain is near zero both for an internalized skill and when neither the model nor the skill delivers the capability, a state we call uncovered. | result 0.45 | 0.51 | 0.12 | asserted 1.00 |
| We instead withhold the skill, score the model on a task synthesized from it, and trust a high unaided score only when another model gains from the skill, since nearly half of high scores come from tasks no model benefits from. | method 0.90 | 0.86 | 0.38 | asserted 0.97 |
| Under four models, a published gain rule applied with one task per skill removes over half of the uncovered skills. | result 1.00 | 0.69 | 0.16 | asserted 1.00 |
| On SkillsBench, 28 of its 31 retirements rest on probes no run passed in full, while retiring on the unaided score costs no detectable reward. | result 1.00 | 0.85 | 0.19 | asserted 1.00 |
| Unaided verdicts reproduce across tasks more often than the gain rule's (Cohen's kappa 0.42 against 0.23) but differ across models, so a hub must read them per model. | result 0.90 | 0.91 | 0.18 | asserted 1.00 |
| We release 1,251 skills with tasks and item scores under four models. | contribution 0.95 | 0.94 | 0.04 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 0.56 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-26: 1 request, 6,972 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (3)
References (41)
- [1] Ziv Bar-Yossef, Andrei Z. Broder, Ravi Kumar, and Andrew Tomkins. 2004. Sic transit gloria telae: towards an understanding of the Web’s decay. InProceedings of the 13th International Conference on World Wide Web. ACM, New York, NY, USA, 328–337. doi:10.1145/988672.988716 on Pithdoi:10.1145/988672.988716
- [2] Robert L. Brennan. 2001.Generalizability Theory. Springer, New York. doi:10. 1007/978-1-4757-3456-0 on Pith
- [3] David Carmel, Doron Cohen, Ronald Fagin, Eitan Farchi, Michael Herscovici, Yoelle S. Maarek, and Aya Soffer. 2001. Static index pruning for information retrieval systems. InProceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Informa... on Pithdoi:10.1145/383952.383958
- [4] Filipe R. Cogo, Gustavo A. Oliva, and Ahmed E. Hassan. 2022. Deprecation of Packages and Releases in Software Ecosystems: A Case Study on npm.IEEE Transactions on Software Engineering48, 7 (2022), 2208–2223. doi:10.1109/TSE. 2021.3055123 on Pithdoi:10.1109/tse
- [5] Lee J. Cronbach and Lita Furby. 1970. How we should measure "change": Or should we?Psychological Bulletin74, 1 (1970), 68–80. doi:10.1037/h0029382 on Pithdoi:10.1037/h0029382
- [6] Kexin Ding, Yang Zhou, Can Jin, Feng Tong, Mu Zhou, and Dimitris N. Metaxas on Pith
- [7] Gen Dong, Yanjie Gao, Liqun Li, Tianyin Xu, Yu Hua, and Fan Yang. 2026. Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents. arXiv:2608.11888 [cs.AI] https://arxiv.org/abs/2608.11888 on PithPith's reviewarXiv
- [8] Jeffrey R. Edwards. 2001. Ten Difference Score Myths.Organizational Research Methods4, 3 (2001), 265–287. doi:10.1177/109442810143005 on Pithdoi:10.1177/109442810143005
- [9] Amirata Ghorbani and James Zou. 2019. Data Shapley: Equitable valuation of data for machine learning. InProceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97). PMLR, 2242–2251 on Pith
- [10] Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. 2026. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering? arXiv:2603.15401 [cs.SE] https://arxiv.org/ abs/2603.15401 on PithPith's reviewarXiv
- [11] Paul W. Holland and Donald B. Rubin. 1982. On Lord’s paradox.ETS Research Report Series1982, 2 (1982). doi:10.1002/j.2333-8504.1982.tb01321.x on Pithdoi:10.1002/j.2333-8504.1982.tb01321.x
- [12] Wentao Hu, Zhendong Chu, Yiming Zhang, Junda Wu, Ming Jin, Xiangyu Zhao, Yilei Shao, Yanfeng Wang, and Qingsong Wen. 2026. SkillBrew: Multi-Objective Curation of Skill Banks for LLM Agents. arXiv:2605.29440 [cs.CL] https://arxiv. org/abs/2605.29440 on PithPith's reviewarXiv
- [13] Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju, Subodh Prabhu, Yogesh Dangi, Wei Luo, and Seong Hee Lee. 2026. Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills... on PithPith's reviewarXiv
- [14] Cory Koedel, Kata Mihaly, and Jonah E. Rockoff. 2015. Value-added modeling: A review.Economics of Education Review47 (2015), 180–195. doi:10.1016/j. econedurev.2015.01.006 on Pithdoi:10.1016/j
- [15] Xiangyi Li, Yimin Liu, Wenbo Chen, Bingran You, Zonglin Di, Yifeng He, Sheng- han Zheng, Kyoung Whan Choe, Jiankai Sun, Shuyi Wang, Chujun Tao, Binxu Li, Xuandong Zhao, Hejia Geng, Xiaojun Wu, Junwei Zhou, Xiaokun Chen, Han- wen Xing, Yubo Li, Qunhong Zeng, Di Wang, Yuanli Wan... on PithPith's reviewarXiv
- [16] Yubo Li. 2026. Dynamic Agent Skills: A Lifecycle Survey and Taxonomy of Evolving Skill Libraries.Transactions on Machine Learning Research(2026). arXiv:2607.10113 https://arxiv.org/abs/2607.10113 on PithPith's reviewarXiv
- [17] Hongxiang Lin, Zhirui Kuai, Erpeng Xue, and Lei Wang. 2026. SKILLC: Learn- ing Autonomous Skill Internalization in LLM Agents via Contrastive Credit Assignment. arXiv:2605.27899 [cs.AI] https://arxiv.org/abs/2605.27899 on PithPith's reviewarXiv
- [18] Hongyi Liu, Haoyan Yang, Tao Jiang, Bo Tang, Feiyu Xiong, Yuyu Luo, and Zhiyu Li. 2026. SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution. arXiv:2605.18401 [cs.CL] https://arxiv.org/abs/ 2605.18401 on PithPith's reviewarXiv
- [19] Frederic M. Lord. 1967. A paradox in the interpretation of group comparisons. Psychological Bulletin68, 5 (1967), 304–305. doi:10.1037/h0025105 on Pithdoi:10.1037/h0025105
- [20] Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. 2026. SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization. arXiv:2604.02268 [cs.LG] https://arxiv.org/abs/2604.02268 on PithPith's reviewarXiv
show 21 more
- [21] Daniel F. McCaffrey, J. R. Lockwood, Daniel M. Koretz, and Laura S. Hamilton. 2003.Evaluating Value-Added Models for Teacher Accountability. Monograph MG-158. RAND Corporation, Santa Monica, CA. doi:10.1037/e658712010-001 on Pithdoi:10.1037/e658712010-001
- [22] Johannes Moll, Jean-Philippe Corbeil, Jiazhen Pan, Martin Hadamitzky, Daniel Rueckert, Lisa Adams, and Keno Bressem. 2026. GRASP: Gated Regression- Aware Skill Proposer for Self-Improving LLM Agents. InProceedings of the 2026 Conference on Empirical Methods in Natural Language... on PithPith's reviewarXiv
- [23] Michael Nguyen, Quoc Nguyen, and Paul Vuong. 2026. Recursive Self-Evolving Agents via Held-Out Selection. arXiv:2606.28374 [cs.AI] https://arxiv.org/abs/ 2606.28374 on PithPith's reviewarXiv
- [24] Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zha, Maohao Shen, Vishy Tirumalashetty, George Lee, Jiawei Han, Tomas Pfister, and Chen- Yu Lee. 2026. SkillOS: Learning Skill Curation for Self-Evolvin... on PithPith's reviewarXiv
- [25] Hongji Pu, Xinyuan Song, and Liang Zhao. 2026. SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems. arXiv:2605.13716 [cs.SE] https://arxiv.org/abs/2605.13716 on PithPith's reviewarXiv
- [26] Mark D. Reckase. 1983. A procedure for decision making using tailored testing. InNew Horizons in Testing, David J. Weiss (Ed.). Academic Press, 237–255. doi:10. 1016/b978-0-12-742780-5.50023-8 on Pith
- [27] Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich, Maria I. Gorinova, Amy Heineike, and Rob Willoughby. 2026. A Framework for Evaluating Agentic Skills at Scale. arXiv:2606.17819 [cs.SE] https://arxiv.org/abs/2606.17819 on PithPith's reviewarXiv
- [28] Junhao Shen, Teng Zhang, Xiaoyan Zhao, and Hong Cheng. 2026. Dy- namic Skill Lifecycle Management for Agentic Reinforcement Learning. arXiv:2605.10923 [cs.LG] https://arxiv.org/abs/2605.10923 on PithPith's reviewarXiv
- [29] Boyin Tan, Xiaowei Huang, and Youcheng Sun. 2026. Skill Coverage: A Test Adequacy Metric for Agent Skills. arXiv:2606.20659 [cs.AI] https://arxiv.org/ abs/2606.20659 on PithPith's reviewarXiv
- [30] Darshan Tank and Baran Nama. 2026. The Regression Tax: Decomposing Why Skills Help and Hurt LLM Agents. arXiv:2607.22520 [cs.AI] https://arxiv.org/ abs/2607.22520 on PithPith's reviewarXiv
- [31] Abraham Wald. 1945. Sequential tests of statistical hypotheses.The Annals of Mathematical Statistics16, 2 (1945), 117–186. doi:10.1214/aoms/1177731118 on Pithdoi:10.1214/aoms/1177731118
- [32] Leye Wang, Zixing Wang, and Anjie Xu. 2026. SkillTester: Benchmarking Utility and Security of Agent Skills. arXiv:2603.28815 [cs.CR] https://arxiv.org/abs/ 2603.28815 on PitharXiv
- [33] Yanze Wang, Pengfei Yao, Tianyi Sun, Chuanrui Hu, Yan Xiao, Xiaotian Luo, Yunyun Han, Yifan Chen, Jun Sun, and Yafeng Deng. 2026. SkillCorpus: Consol- idating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents. arXiv:2607.15557 [cs.CL] https://arxiv.org/abs/2607.15557 on PithPith's reviewarXiv
- [34] Yixuan Wang, Yiyang Zhou, Yiming Liang, Congyu Zhang, Fuxiao Liu, Jiawei Zhou, and Huaxiu Yao. 2026. Not All Skills Help: Measuring and Repairing Agent Knowledge. arXiv:2606.15390 [cs.CL] https://arxiv.org/abs/2606.15390 on PitharXiv
- [35] Jianxiang Yu, Jiapeng Zhu, Bochen Lin, Qier Cui, Zichen Ding, and Xiang Li. 2026. Skill is Not One-Size-Fits-All: Model-Aware Skill Alignment for LLM Agents. arXiv:2605.30723 [cs.CL] https://arxiv.org/abs/2605.30723 on PithPith's reviewarXiv
- [36] Zhonghao Zhan, Xiao Ma, and Hamed Haddadi. 2026. Safe Skill Retirement for Physical Agents. arXiv:2609.29543 [cs.AI] https://arxiv.org/abs/2609.29543 on PithPith's reviewarXiv
- [37] Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, and Peiyang He. 2026. The Blind Curator: How a Biased Judge Silently Disables Skill Retirement in Self-Evolving Agents. arXiv:2607.07436 [cs.AI] https://arxiv.org/ abs/2607.07436 on PithPith's reviewarXiv
- [38] Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, and Peiyang He. 2026. Library Drift: Diagnosing and Fixing a Silent Failure Mode in Self-Evolving LLM Skill Libraries. InICML 2026 Workshop on Failure Modes in Agentic AI. Seoul, South Korea. arXiv:2605.19576 on PithPith's reviewarXiv
- [39] Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, and Peiyang He. 2026. Ratchet: How Reliable Must an LLM Judge Be to Retire a Skill? arXiv:2605.22148 [cs.AI] https://arxiv.org/abs/2605.22148 on PithPith's reviewarXiv
- [40] Yangqiaoyu Zhou, Mohammad Alqudah, Kwei-Herng Lai, Aaron Halfaker, Yingqi Xiong, and Yaar Harari. 2026. A Single Rewrite Suffices: Empirical Lessons from Production Skill Description Optimization. arXiv:2606.30775 [cs.CL] https: //arxiv.org/abs/2606.30775 Reproducibility state... on PithPith's reviewarXiv
- [2026] Agent Skill Evaluation and Evolution: Frameworks and Benchmarks. arXiv:2606.11435 [cs.CL] https://arxiv.org/abs/2606.11435 on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.



No comments yet.
Sign in to comment