Artificial Intelligence · arXiv:2608.21964 v1 · submitted 22 Aug 2026 · 46 references
Repo2Skill-Evo: Repository Skills Go Stale in Silence
The paper says
“Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects.”
more of the abstract
Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Aug 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A genuinely useful benchmark for a real maintenance problem, with one load-bearing annotation-quality question that needs to be answered before trusting the exact numbers. This line is wrong
2 major objections · 3 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 46 references say.
The paper says, at Appendix B.2
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The headline claim—six frontier agents at 29.9%–69.7% avg@3 macro F1—is computed as an exact line-level match against G_delta. Appendix B.3 reports that G_delta is obtained by automated candidate discovery followed by manual verification, with verification also deciding which 'semantically expressed claims' and 'dependent instructions' are invalidated by the patch.
more of objection 1
No inter-annotator agreement, annotation guidelines, or objective decision rule for including dependent lines are reported, and the text does not state that the gold files themselves are released (Appendix B says gold annotations are stored separately, and Appendix D.4 says released materials include per-transition assignments, but not G_delta). Because every model's precision, recall, and F1 are computed against exactly these line judgments, systematic disagreement among annotators would recalibrate all reported scores and could change the model ordering. The paper's claim that the evaluation is 'strictly patch-grounded' is weakened by the subjective component of deciding which dependent instructions are stale. Please release G_delta, publish the verification guidelines, report agreement statistics on an independently re-annotated sample, and provide a sensitivity analysis of the headline F1 under plausible variations in dependent-line decisions.
The paper says, at Appendix B.2
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The claimed isolation of SV1 from the release patch and gold labels is a procedural assertion rather than an auditable property of the released materials. The text states that both candidate skill sets are generated from the pre-release repository alone and that the expert revision draws on the same V1 evidence, so the release patch and gold labels remain outside the construction of SV1 'from end to end.'
more of objection 2
Because SV1 is the common input to every maintenance agent and G_delta is defined relative to SV1, any undetected leakage of V2 knowledge into SV1 would co-construct the task and affect all six model scores. To make this load-bearing assumption checkable, the paper should release the candidate-generation transcripts, expert revision diffs, and the verification checklist used against the V1 repository, or describe an independent re-derivation check of a sample of skill sets.
3 minor objections
The paper says, at §4.2 and Appendix A
Pith says: minor objection 1 of 3
The motivating utility study uses manually scored rubric judgments with no inter-rater reliability statistics; given the small sample of ten repositories, the gains should be explicitly labeled as preliminary or accompanied by scoring reliability evidence.
The paper says, at §3.1
Pith says: minor objection 2 of 3
The grounding gate thresholds—'minimum reference count' and the 'preset threshold' on unverifiable references—are not specified; please report the values and, ideally, a brief sensitivity analysis.
show 1 more
The paper says, at §3.4
Pith says: minor objection 3 of 3
Macro precision excludes transitions for which precision is undefined in all three runs, but the number of excluded transitions is not reported; please state this count so readers can assess the potential upward bias in the reported macro precision.
What would settle it
Select 20 transitions stratified by difficulty and |G_delta|. Have two independent annotators, blind to original labels and to all agent outputs, re-derive G_delta from the same SV1 and official V1→V2 patch using a written decision rule. Compute line-level Cohen's kappa between annotators and between each annotator and the released G_delta. Recompute all six models' avg@3 macro R/P/F1 against each independently re-derived set. If line-level kappa is below 0.8, or if any model's corpus-level F1 moves by more than 5 points, the headline range and model ordering are not robust to gold-label uncertainty and the benchmark needs revision. Report both kappas and the recalibrated score table.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Aug 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 3 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Key thing to know: this paper turns the maintenance of repository-specific agent skills into a concrete benchmark, and its core finding — six frontier agents top out at 69.7% avg@3 macro F1 on patch-grounded stale-line removal — survives a close read. The framing of repository skills as versioned knowledge assets is useful, and the authors did the work to ground the evaluation in official release patches rather than model outputs.
What is actually new: the release-conditioned maintenance task, 105 official transitions across 57 repos, a line-level gold obsolete set derived from official patches, and a patch-grounded removal metric that penalizes both missed stale content and over-editing. The oracle localization ablation is a good addition and shows localization matters but is not sufficient. The results are reported with repository-clustered bootstrap CIs, and the metric has no fitted parameters. The utility study is small but only used as motivation. Prior work on skill maintenance (SkillGuard, SkillOps, Library Drift, SkillHone) is covered; I don't see a citation gap that matters.
The soft spot is exactly the one the stress-test highlights. G_delta is the load-bearing evaluation target, and it rests on automated candidate discovery plus manual line-level verification, with no inter-annotator agreement, no annotation guidelines, and no independent audit. Appendix B.3 admits that deciding which 'semantically expressed claims' and 'dependent instructions' are invalidated by the patch is a manual judgment. Since recall and precision are exact line-level matches against G_delta, label noise would recalibrate every reported F1. The qualitative conclusion likely survives — these agents are not reliably maintaining skills — but the precise 29.9–69.7 range and model ordering are not yet on firm footing. The same-team construction of SV1 and the gold set is also a procedural claim, not an auditable one, until the data are released. The paper says materials will be released, but the preprint does not make them accessible.
Minor concerns: the corpus is deliberately staleness-positive (acknowledged), and the NL Judge is LLM-based and complementary rather than primary. Neither changes my view.
Bottom line: this is a serious benchmark paper aimed at the agent-skill and LLM-software-maintenance community. I would send it to peer review, but I would require two things before treating the numbers as a stable reference: independent annotation agreement on a sample of G_delta, and public release of the gold files and skill sets. With those, I'd cite it.
The strongest objection
The headline claim — that six frontier agents reach only 29.9%–69.7% avg@3 macro F1 — is computed against a single human-constructed gold obsolete set G_delta (Section 3.4). Appendix B.3 states that G_delta is built by automated candidate discovery followed by manual verification against direct patch evidence, where verification also decides which 'semantically expressed claims' and 'dependent instructions' are invalidated by the patch. No inter-annotator agreement, no objective decision rule, and no annotation guidelines are reported, and the gold files are not part of the artifacts described as released with the preprint. Because recall and precision are line-level exact matches against G_delta, any systematic disagreement about which dependent lines are stale directly changes every model's recall, precision, and F1. The reported model ordering and the 29.9–69.7 range could shift if independent annotators label the same patches differently. The risk is amplified by a shared-process design: the same team selects and revises SV1 (Appendix B.2) and then constructs G_delta, so the asserted isolation of SV1 from the release patch is a procedural claim rather than an auditable property of the released materials. The central claim therefore depends on G_delta being both accurate and reproducible, and that condition is currently the least secure link in the argument.
The reader's weakest_assumption identifies exactly this: G_delta is manually constructed with no inter-annotator agreement or objective rule, and every reported precision and recall number is miscalibrated if those judgments are biased. I agree. This is the single most load-bearing concern because every headline number filters through line-level labels in G_delta. Other plausible objections — the staleness-positive corpus, the lack of downstream reruns of maintained skills, and the minimal scaffold — are either explicitly acknowledged as limitations (Section 5) or secondary to the headline metric. The paper has real independent support: a fixed SV1, an oracle localization ablation, repository-clustered confidence intervals, and a patch-grounded evaluation design. If independent re-annotation reproduces G_delta with high agreement and the six models' scores shift minimally, the central claim very likely holds. But until that check is run, the benchmark cannot serve as a stable reference, and the conditional verdict is appropriate. Since the reader already returned CONDITIONAL, my read does not change the verdict; it confirms the condition that needs to be met.
The referee's report
If the headline measurement is accepted, Repo2Skill-Evo is a valuable and relatively clean benchmark. Its strengths include grounding staleness in official release patches rather than in model outputs, separating skill maintenance from code migration, reporting repository-clustered bootstrap confidence intervals and paired differences for the top-model ordering, and including an oracle localization ablation that sharpens the bottleneck diagnosis. The distinction between avg@3 and best@3, the fixed shared scaffold, and the release of version tags and commit hashes are also to the paper's credit. The practical conclusion—that externalized repository knowledge can become silently stale and that even frontier agents cannot reliably maintain it—is important for the emerging skill-library ecosystem. The central caveat is that the benchmark's gold obsolete set is built by manual annotation whose reliability and auditability are not yet demonstrated; the headline numbers depend directly on that annotation.
Pith's reading of the paper (a machine's summary, not the authors')
This paper argues that repository skills — the reusable instruction packages that tell an LLM agent which APIs to call, which scripts to run, and which conventions a codebase expects — decay silently when the repository releases a new version. A skill written for version 1 stays loadable and retrievable after version 2 ships, so nothing flags that its guidance is now obsolete, and agents keep retrieving and following it. To make this decay measurable, the paper introduces Repo2Skill-Evo, a benchmark of 105 official release transitions across 57 repositories in which an agent must update a frozen V1 skill set against the official V1-to-V2 patch, removing stale lines the patch verifies as obsolete while preserving valid ones. The central empirical claim is that frontier agents cannot do this reliably: six leading models score only 29.9%–69.7% avg@3 macro F1 under a patch-grounded removal metric, with failures split between missing affected skill files and editing too broadly once files are found. A careful reader should care because the same independence that makes skills cheap to reuse makes their decay invisible, so any agent relying on such skills inherits outdated guidance with no warning.
What carries the argument. The load-bearing object is the patch-grounded gold obsolete set $G_\Delta$: the set of V1 skill lines that the official V1-to-V2 release patch directly shows to be no longer valid under V2, built by automated candidate discovery followed by line-level manual verification against patch evidence. The removal metric compares the agent's changed-line set $R$ (V1 skill lines on the deletion side of the $S_{V1} \to S_{V2}$ diff) with $G_\Delta$ through recall $|G_\Delta \cap R|/|G_\Delta|$, precision $|G_\Delta \cap R|/|R|$, and their harmonic mean, so a good score requires removing exactly the patch-verified stale lines and nothing else. On the construction side, Repo2Skill produces the frozen V1 skill set by staged distillation with a grounding gate that verifies every cited symbol and path against a static index $I_{\mathrm{sym}}$, guaranteeing source traceability, and the maintenance scaffold is a deliberately minimal single-agent environment with generic shell, file, and editing tools under a 50-turn budget. Scoring only the V1-side removal — never matching newly written replacement lines — is what makes the headline result auditable, since a valid V2 replacement admits many equivalent forms; the complementary five-dimensional NL Judge (API, Files, Info, Prompt, NL) assesses the final state's semantic correctness.
Core claim. The paper establishes silent staleness as a distinct, release-driven failure mode of repository-grounded agent skills and shows empirically that it is currently unmanaged. On Repo2Skill-Evo — 105 selected official release transitions across 57 actively maintained repositories, each pairing a fixed, source-traceable V1 skill set with a patch-verified gold obsolete set $G_\Delta$ (median 92 stale lines) — six frontier agents (Claude-opus-4.6, GLM-5.1, GPT-5.4, Kimi-K2.5, Doubao-Seed2-pro, MiniMax-M2.5) achieve only 29.9%–69.7% avg@3 macro F1. Every evaluated transition invalidates part of the V1 skill set, and even the strongest model leaves a large fraction of patch-verified stale content in place or edits valid content off-target. Analysis of 1,890 runs identifies two complementary bottlenecks: incomplete localization of affected skill files, whose run-level correlation with F1 is $r = 0.650$, and imperfect edit selection, since GPT-5.4 reaches the highest recall (74.6%) but low precision (55.7%) by removing 2.84 times the gold removal target. Oracle file-level localization on the hardest-20 transitions raises mean F1 from 32.8% to 45.0%, yet the best model still reaches only 62.3%, so localization alone is insufficient. A motivating utility study shows the stakes: fixed V1 skills raise task utility from 5.88 to 8.68 on ten sampled repositories, with the largest gains (+3.98) on low-baseline repositories, meaning the knowledge that goes stale is exactly the knowledge agents most need supplied externally.
The premise it rests on. The gold obsolete set $G_\Delta$ is built by human judgment about which V1 skill lines the release patch invalidates, including semantically expressed claims and dependent instructions, and the paper reports no inter-annotator agreement or objective decision rule; if those judgments are biased, every recall, precision, and F1 number in the paper is miscalibrated.
If the paper is right (Pith's inferences, not the authors')
Repository skills should be treated as versioned knowledge assets: each skill set must carry the release it was distilled from and be re-validated when the repository moves to a new release, because staleness produces no explicit error.
Maintenance quality is bottlenecked by two separable capabilities — finding the affected skill files and choosing exactly which lines to edit — so improving either axis should translate directly into F1 gains, with the oracle experiment bounding the localization axis at roughly +12 mean F1 points on the hardest transitions.
Repeated attempts do not close the gap: best@3 exceeds avg@3 by 6.4–16.1 points across models, yet tops out at 76.1% F1, so the shortfall is not merely run-to-run noise.
The benchmark provides a fixed target on which future maintenance systems — contract checkers, drift detectors, revision harnesses — can be scored, with the strongest current agent leaving roughly 30% of patch-verified stale content in place.
Editorial inference: the same ingredients — the symbol index, the release patch, and the skill set — could be combined into a retrieval-time staleness alarm that diffs a retrieved skill line against the current release before an agent follows it; the paper does not build this, but its machinery supplies the parts.
Editorial inference: because the gold set rests on single-annotator judgment, the true maintainability headroom is uncertain; a replication with multiple annotators and agreement statistics would reveal how much of the 29.9%–69.7% range is metric artifact rather than model limitation.
Editorial inference: the paper's implicit claim that stale skills cause real harm could be tested by running the agents' produced V2 skill sets through the same utility rubric used in the motivating study instead of judging them only against gold removals; the paper explicitly defers this downstream check.
Editorial inference: the low-baseline utility result hints that maintenance effort should be prioritized where agents have least prior knowledge, so a deployment heuristic weighting repositories by baseline utility or staleness density may be nearly as valuable as further model improvements.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. What remains unclear is whether that improvement is durable. The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. Externalizing knowledge into a skill can therefore make its decay invisible. We study whether agents can keep this externalized knowledge current. Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“Repository skills go stale in silence, and even frontier agents cannot reliably maintain them.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.37 · human participants 0.04 · controlled comparison 0.11 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.48 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Large language model (LLM) agents increasingly operate over evolving software repositories, where success depends on repository-specific procedural knowledge: which APIs to call, which scripts to run, and which conventions the current release expects. | background 1.00 | 0.06 | 0.02 | asserted 0.99 |
| Agent skills externalize this knowledge into reusable units, and prior work shows that they can improve agent performance. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| What remains unclear is whether that improvement is durable. | gap 1.00 | 0.05 | 0.03 | unclear 0.64 |
| The same version specificity that makes a skill useful also makes it fragile: after a release, it may become stale without raising any explicit signal, while continuing to provide obsolete guidance. | background 0.84 | 0.10 | 0.09 | hedged possibility 0.96 |
| Externalizing knowledge into a skill can therefore make its decay invisible. | background 0.65 | 0.11 | 0.12 | asserted 0.58 |
| We study whether agents can keep this externalized knowledge current. | contribution 0.73 | 0.33 | 0.03 | unclear 0.83 |
| Repo2Skill-Evo casts each release transition as a skill-maintenance task: given a V1 skill set and the official V1-to-V2 patch, an agent must update obsolete skill content while preserving guidance that remains valid. | method 0.80 | 0.78 | 0.04 | asserted 0.95 |
| Across 57 real-world repositories and 105 selected release transitions, every evaluated transition invalidates part of the V1 skill set. | result 0.98 | 0.94 | 0.12 | asserted 1.00 |
| Yet six frontier agents reach only 29.9%-69.7% avg@3 macro F1 under a patch-grounded removal metric that balances stale-content recall against over-editing precision. | result 1.00 | 0.93 | 0.15 | asserted 1.00 |
| Across runs, two opposing errors dominate: incomplete coverage of affected files in the skill set leaves stale content untouched, while overbroad editing is associated with higher recall but lower precision. | result 0.89 | 0.90 | 0.08 | asserted 1.00 |
| Repository skills go stale in silence, and even frontier agents cannot reliably maintain them. | interpretation 0.75 | 0.85 | 0.94 | asserted 1.00 |
What kind of paper the classifier reports this is: empirical study 0.92 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 8,178 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
References (46)
- [1] Agent Skills. Agent Skills overview, 2026. URLhttps://agentskills.io/home. [Accessed 04-07-2026] on Pith
- [2] Hussein Alrubaye, Mohamed Wiem Mkaouer, and Ali Ouni. Migrationminer: An automated detection tool of third-party java library migration at the method level. In2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, Cleveland, OH, USA, September 29... on PitharXiv
- [3] Anthropic. System card: Claude Opus 4.6, 2026. URL https://www.anthropic.com/ claude-opus-4-6-system-card. [Accessed 04-07-2026] on Pith
- [4] Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, and Bing Zhao. SWE-CI: evaluating agent capabilities in maintaining codebases via continuous integration.CoRR, abs/2603.03823, 2026. doi: 10.48550/ARXIV.2603.03823. URLhttps://doi.org/10.48550/arXiv.2603.03823 on Pithdoi:10.48550/arxiv.2603.03823
- [5] Linfeng Fan, Yuan Tian, Ziwei Li, and Zhiwu Lu. Skill drift is contract violation: Proactive maintenance for LLM agent skill libraries.CoRR, abs/2605.10990, 2026. doi: 10.48550/ARXIV.2605.10990. URL https: //doi.org/10.48550/arXiv.2605.10990 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.10990
- [6] Haoyu Gao, Jai Lal Lulla, Hong Yi Lin, Sebastian Baltes, Christoph Treude, and Mansooreh Zahedi. From registry to repository: How ai agent skills are written, adapted, and maintained, 2026. URLhttps://arxiv.org/ abs/2607.00911 on PithPith's reviewarXiv
- [7] GLM-Team. GLM-5: from vibe coding to agentic engineering.CoRR, abs/2602.15763, 2026. doi: 10.48550/ ARXIV.2602.15763. URLhttps://doi.org/10.48550/arXiv.2602.15763 on PithPith's reviewarXivdoi:10.48550/arxiv.2602.15763
- [8] Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. Swe-skills- bench: Do agent skills actually help in real-world software engineering?CoRR, abs/2603.15401, 2026. doi: 10.48550/ARXIV.2603.15401. URLhttps://doi.org/10.48550/arXiv.2603.15401 on PithPith's reviewdoi:10.48550/arxiv.2603.15401
- [9] Mohayeminul Islam, Ajay Kumar Jha, Ildar Akhmetov, and Sarah Nadi. Characterizing python library migrations. Proc. ACM Softw. Eng., 1(FSE):92–114, 2024. doi: 10.1145/3643731. URL https://doi.org/10.1145/3643731 on Pithdoi:10.1145/3643731
- [10] Yanna Jiang, Delong Li, Haiyu Deng, Baihe Ma, Xu Wang, Qin Wang, and Guangsheng Yu. Sok: Agentic skills - beyond tool use in LLM agents.CoRR, abs/2602.20867, 2026. doi: 10.48550/ARXIV.2602.20867. URL https://doi.org/10.48550/arXiv.2602.20867 on PithPith's reviewarXivdoi:10.48550/arxiv.2602.20867
- [11] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. Swe-bench: Can language models resolve real-world github issues? InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-... on Pith
- [12] Seungone Kim, Jamin Shin, Yejin Choi, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. InThe Twelfth International Conference on Learning ... on Pith
- [13] Raula Gaikovina Kula, Ali Ouni, Daniel M. Germán, and Katsuro Inoue. An empirical study on the impact of refactoring activities on evolving client-used apis.Inf. Softw. Technol., 93:186–199, 2018. doi: 10.1016/J.INFSOF. 2017.09.007. URLhttps://doi.org/10.1016/j.infsof.2017.09.007 on Pithdoi:10.1016/j.infsof
- [14] Man Ho Lam, Chaozheng Wang, Hange Liu, Jingyu Xiao, Haau-sing Li, Jen-tse Huang, Terry Yue Zhuo, and Michael R. Lyu. Swe-chain: Benchmarking coding agents on chained release-level package upgrades.CoRR, abs/2605.14415, 2026. doi: 10.48550/ARXIV.2605.14415. URLhttps://doi.org/1... on PithPith's reviewarXivdoi:10.48550/arxiv.2605.14415
- [15] Hanyu Li, Yichi Zhang, Speed Zhu, Hang Su, Jun Zhu, and Yinpeng Dong. Repomirage: Probing repository context reasoning in code agents with perturbations.CoRR, abs/2605.26177, 2026. doi: 10.48550/ARXIV.2605.26177. URLhttps://doi.org/10.48550/arXiv.2605.26177 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.26177
- [16] Xiangyi Li, Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, ... on PithPith's reviewarXivdoi:10.48550/arxiv.2602.12670
- [17] Zhiwei Li and Yong Hu. Skillhone: A harness for continual agent skill evolution through persistent decision history. CoRR, abs/2606.08671, 2026. doi: 10.48550/ARXIV.2606.08671. URLhttps://doi.org/10.48550/arXiv.2606. 08671 on PithPith's reviewarXivdoi:10.48550/arxiv.2606.08671
- [18] Tianyang Liu, Canwen Xu, and Julian J. McAuley. Repobench: Benchmarking repository-level code auto- completion systems. InThe Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URLhttps://openreview.n... on Pith
- [19] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:... on Pith
- [20] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: NLG evaluation using gpt-4withbetterhumanalignment. InHoudaBouamor, JuanPino, andKalikaBali, editors,Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EM... on Pithdoi:10.18653/v1/2023.emnlp-main.153
show 26 more
- [21] Yuxuan Liu, Zhaochen Su, Lingyun Xie, Yuhao Zhang, Qing Zong, Jiahe Guo, Zhongwei Xie, Yiyan Ji, Yauwai Yim, Hongyu Luo, Xiyu Ren, Ruan Chenyu, Haoran Li, and Yangqiu Song. Skillrevise: Improving LLM-authored agent skills via trace-conditioned skill revision.CoRR, abs/2606.011... on PithPith's reviewarXivdoi:10.48550/arxiv.2606.01139
- [22] Zhengxi Lu, Zhiyuan Yao, Jinyang Wu, Chengcheng Han, Qi Gu, Xunliang Cai, Weiming Lu, Jun Xiao, Yueting Zhuang, and Yongliang Shen. SKILL0: in-context agentic reinforcement learning for skill internalization.CoRR, abs/2604.02268, 2026. doi: 10.48550/ARXIV.2604.02268. URLhttps:... on PithPith's reviewarXivdoi:10.48550/arxiv.2604.02268
- [23] Zijian Lu, Yiping Zuo, Yupeng Nie, Xin He, Weibei Fan, Lianyong Qi, and Shi Jin. Contractskill: Repairable contract-based skills for multimodal web agents.CoRR, abs/2603.20340, 2026. doi: 10.48550/ARXIV.2603.20340. URLhttps://doi.org/10.48550/arXiv.2603.20340 on PithPith's reviewarXivdoi:10.48550/arxiv.2603.20340
- [24] MiniMax. MiniMax-M2.5 model card, 2026. URLhttps://huggingface.co/MiniMaxAI/MiniMax-M2.5. [Accessed 04-07-2026] on Pith
- [25] OpenAI. GPT-5.4 thinking system card, 2026. URL https://openai.com/index/ gpt-5-4-thinking-system-card/. [Accessed 04-07-2026] on Pith
- [26] Hongji Pu, Xinyuan Song, and Liang Zhao. Skillops: Managing LLM agent skill libraries as self-maintaining software ecosystems.CoRR, abs/2605.13716, 2026. doi: 10.48550/ARXIV.2605.13716. URLhttps://doi.org/ 10.48550/arXiv.2605.13716 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.13716
- [27] PyTorch. Repository agent skills in the PyTorch codebase, 2026. URLhttps://github.com/pytorch/pytorch/ tree/main/.claude/skills. Public skill directory maintained under the repository’s.claude/skills/ path. [Accessed 04-07-2026] on Pith
- [28] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Har... on Pith
- [29] Bytedance Seed. Seed2.0 model card: Towards intelligence frontier for real-world complexity.CoRR, abs/2607.00248, 2026. doi: 10.48550/ARXIV.2607.00248. URLhttps://doi.org/10.48550/arXiv.2607.00248 on PithPith's reviewarXivdoi:10.48550/arxiv.2607.00248
- [30] Kimi Team. Kimi K2.5: visual agentic intelligence.CoRR, abs/2602.02276, 2026. doi: 10.48550/ARXIV.2602.02276. URLhttps://doi.org/10.48550/arXiv.2602.02276 on PithPith's reviewarXivdoi:10.48550/arxiv.2602.02276
- [31] Cédric Teyton, Jean-Rémy Falleri, Marc Palyart, and Xavier Blanc. A study of library migrations in java.J. Softw. Evol. Process., 26(11):1030–1052, 2014. doi: 10.1002/SMR.1660. URLhttps://doi.org/10.1002/smr.1660. 15 on Pithdoi:10.1002/smr.1660
- [32] Minh V. T. Thai, Tue Le, Dung Nguyen Manh, Huy Phan Nhat, and Nghi D. Q. Bui. SWE-EVO: benchmarking coding agents in long-horizon software evolution scenarios.CoRR, abs/2512.18470, 2025. doi: 10.48550/ARXIV. 2512.18470. URLhttps://doi.org/10.48550/arXiv.2512.18470 on PithPith's reviewarXivdoi:10.48550/arxiv.2512.18470
- [33] Fengcai Wen, Csaba Nagy, Gabriele Bavota, and Michele Lanza. A large-scale empirical study on code-comment inconsistencies. In Yann-Gaël Guéhéneuc, Foutse Khomh, and Federica Sarro, editors,Proceedings of the 27th International Conference on Program Comprehension, ICPC 2019, M... on PitharXiv
- [34] Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. Demystifying llm-based software engineering agents.Proc. ACM Softw. Eng., 2(FSE):801–824, 2025. doi: 10.1145/3715754. URL https://doi.org/10.1145/ 3715754 on Pithdoi:10.1145/3715754
- [35] John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. ... on Pith
- [36] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.... on Pith
- [37] Hanrong Zhang, Shicheng Fan, Henry Peng Zou, Yankai Chen, Zhenting Wang, Jiayu Zhou, Chengze Li, Wei- Chieh Huang, Yifei Yao, Kening Zheng, Xue Liu, Xiaoxiao Li, and Philip S. Yu. Coevoskills: Self-evolving agent skills via co-evolutionary verification.CoRR, abs/2604.01687, 20... on PithPith's reviewarXivdoi:10.48550/arxiv.2604.01687
- [38] Xing Zhang, Yanwei Cui, Guanghui Wang, Ziyuan Li, Wei Qiu, Bing Zhu, and Peiyang He. Library drift: Diagnosing and fixing a silent failure mode in self-evolving LLM skill libraries.CoRR, abs/2605.19576, 2026. doi: 10.48550/ARXIV.2605.19576. URLhttps://doi.org/10.48550/arXiv.2605.19576 on PithPith's reviewarXivdoi:10.48550/arxiv.2605.19576
- [39] Yuntong Zhang, Haifeng Ruan, Zhiyu Fan, and Abhik Roychoudhury. Autocoderover: Autonomous program improvement. In Maria Christakis and Michael Pradel, editors,Proceedings of the 33rd ACM SIGSOFT Interna- tional Symposium on Software Testing and Analysis, ISSTA 2024, Vienna, Au... on PitharXiv
- [40] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm- as-a-judge with mt-bench and chatbot arena. In Alice Oh, Tristan Naumann, Amir Globe... on Pith
- [41] Yifan Zhou, Zhentao Zhang, Ziming Cheng, Shuo Zhang, Qizhen Lan, Zhangquan Chen, Zhi Yang, Qianyu Xu, Ronghao Chen, Huacan Wang, and Sen Hu. Skillgenbench: Benchmarking skill generation pipelines for LLM agents.CoRR, abs/2605.18693, 2026. doi: 10.48550/ARXIV.2605.18693. URL ht... on PithPith's reviewarXivdoi:10.48550/arxiv.2605.18693
- [42] Yingli Zhou, Wang Shu, Yaodong Su, Wenchuan Du, Yixiang Fang, and Xuemin Lin. A comprehensive survey on agent skills: Taxonomy, techniques, and applications.CoRR, abs/2605.07358, 2026. doi: 10.48550/ARXIV.2605. 07358. URLhttps://doi.org/10.48550/arXiv.2605.07358. 16 Appendix A... on PithPith's reviewarXivdoi:10.48550/arxiv.2605.07358
- [43] Select release transitions.We identify official releases with publicly available source diffs (Appendix B.1) on Pith
- [44] Materialize release instances.Each transition records its repository,V1 and V2 version tags, commit hashes, and officialV1→V2 release patch on Pith
- [45] Construct the fixed V1 skill set.Repo2Skill produces candidate skill sets from the pre-release reposi- tory, after which an expert selects, revises, and verifies the finalSV1 used by all maintenance agents (Appendix B.2) on Pith
- [46] Construct the gold obsolete set.We identify the V1 skill lines that the release patch directly shows to be invalid underV2, producing the patch-grounded gold setG∆ (Appendix B.3). Transitions withG∆ = ∅ are excluded, yielding the final staleness-positive corpus. Gold annotatio... on Pith
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 12 sentences without a confident label
Compares with 1 paper
-
Unlike skill internalization [ref] , the maintained object remains an external artifact whose validity is tied to a software release.
SKILL0: In-Context Agentic Reinforcement Learning for Skill Internalization · Agent Skill Lifecycle and Maintenance · 0.83
Cites as support 1 paper
-
Extracting this knowledge from scratch for every task is costly and unreliable, since it requires cross-file retrieval and structural understanding that even strong agents do not consistently achieve [ref] .
RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations · Introduction · 0.97
Cites as background 12 papers
-
SkillOps mitigates technical debt via typed contracts and graph-structured maintenance [ref] , while Library Drift investigates retrieval degradation stemming from unregulated expansion within self-evolving skill banks [ref] .
Library Drift: Diagnosing and Fixing a Silent Failure Mode in Self-Evolving LLM Skill Libraries · Agent Skill Lifecycle and Maintenance · 0.92
-
SkillOps mitigates technical debt via typed contracts and graph-structured maintenance [ref] , while Library Drift investigates retrieval degradation stemming from unregulated expansion within self-evolving skill banks [ref] .
SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems · Agent Skill Lifecycle and Maintenance · 0.92
-
Recent agent benchmarks extend from static repair to release-note-derived evolution tasks [ref] , continuous-integration loops [ref] , and chained release-level upgrades [ref] .
SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios · Repository-Level Agents and Software Evolution · 0.86
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment