Pith.

Cryptography and Security · arXiv:2610.25705 v1 · submitted 19 Oct 2026 · 51 references

How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel?

Haoran Xue, Nikta Akbarpour, Gias Uddin, Song Wang

The paper says

“Large language model (LLM) coding agents, such as Claude Code, are increasingly used for security auditing, with task-specific skills providing reusable vulnerability-analysis procedures. However, their effectiveness in detecting real vulnerabilities and the reasons for their failures remain unclear.”

more of the abstract

We conduct the first large-scale empirical study of 37 security skills using Claude Code with four underlying LLMs on 100 real Linux kernel vulnerabilities across ten classes, totaling 14800 runs. We evaluate vulnerability detection at the function, statement, and vulnerability-class levels. Results show that 24.4 percent of runs locate the correct function, 18.3 percent identify both the function and responsible statement, and 15.7 percent correctly classify the vulnerability. Our failure analysis reveals limitations in evidence acquisition, state-relation reasoning, code interpretation, candidate verification, investigation coverage, and reporting. Based on these findings, we propose SKILL-SIF, a self-improving framework that iteratively evolves security skills by learning from missed vulnerabilities and failure patterns. We evolve five top-performing skills using the original benchmark and evaluate them on 100 previously unseen vulnerabilities. SKILL-SIF improves fully correct vulnerability detection by up to eight times (from 2 percent to 16 percent) on unseen vulnerabilities, while aggregate accuracy across four LLMs increases by 1.85 times (from 6.05 percent to 11.20 percent). These findings highlight the limitations of existing security skills and demonstrate the potential of failure-driven skill evolution to improve vulnerability detection in coding agents.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 20 Oct 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

The passes disagreed. Reader: holds up. Skeptic: agrees with the reader. Referee: major revision.

Sobering numbers, solid experimental design, and a practical self-improvement framework. The main results hold up, but the ground-truth annotation deserves a reliability check before publication. This line is wrong

2 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 51 references say.

The paper says, at §3.2, ground-truth construction

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The entire evaluation hinges on the correctness of the ground-truth annotations (vulnerable function, responsible statement, CWE). Section 3.2 describes a human-judgment process that distinguishes vulnerability-relevant operations from supporting changes, but it does not report the number of annotators, the annotation protocol, or any measure of inter-annotator agreement.

more of objection 1

By contrast, §4.3.1 properly reports 77.8% inter-rater agreement and 80.5% human–LLM confirmation for the failure taxonomy, showing that the authors are aware of reliability standards. The same standard should be applied to the ground-truth annotations, which are at least as consequential. Without such a report, the reader cannot assess whether the 11.1% fully correct rate and all derived comparisons are robust to annotation subjectivity. The authors should either present inter-annotator agreement for the ground truth or provide a sensitivity analysis showing that the main findings are stable under plausible annotation variations.

The paper says, at §6.2, Table 6

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The improvement from SKILL-SIF is reported as aggregate fully correct detection rising from 6.05% to 11.20% (Δ=+5.15pp). However, the baseline performance of the five selected skills on the test set varies substantially across models (e.g., Haiku-4.5 starts at 3.4% vs.

more of objection 2

Sonnet-5 at 8.6%). The paper notes this variation (Observation O8) but does not discuss whether the refinement process may inadvertently overfit to the higher-performing models. Given that the refinement model (gpt-5.6-luna) differs from the evaluation LLMs, the concern is limited, but the authors should comment on whether the refinement set’s failure patterns are representative across all four evaluation models, especially the lower-performing ones.

6 minor objections

The paper says, at §6.2

Pith says: minor objection 1 of 6

The paragraph beginning 'After the first revision' contains a nearly identical sentence twice: 'After the first revision, the five skills increase from 24 to 61 fully correct runs. After the first revision, the aggregate number of fully correct runs increases from 24 to 61, a net gain of 37.' Please remove the repetition.

The paper says, at General

Pith says: minor objection 2 of 6

The term 'SKILL-SIF' appears inconsistently (sometimes with mixed-case formatting). Standardize the notation throughout (e.g., SKILL-SIF or SKILL-SIF).

show 4 more

The paper says, at §5.1, Fig. 7

Pith says: minor objection 3 of 6

Figure 7 (framework overview) is described in the text but the caption is minimal. Adding a brief explanatory caption referencing the execute–diagnose–evolve–validate cycle would help readers unfamiliar with the framework.

The paper says, at §3.3, Skill classification

Pith says: minor objection 4 of 6

The paper uses an LLM (Claude Opus-5) to classify skills into PFC and MTC categories, with cross-checking by a second model. While this is a sensible approach for scale, the thresholds for the cross-check ('manually resolve disagreement cases') are not quantified. A brief note on the agreement rate between the two models and the number of human-resolved cases would increase transparency.

The paper says, at §8, Threats to validity

Pith says: minor objection 5 of 6

The threats section is thorough but does not discuss the external validity limitation that the file-guided setting (i.e., providing target file paths) may not reflect unguided auditing scenarios. The paper acknowledges this in §3.5 by stating the evaluation focuses on file-guided identification, but it should be explicitly listed as a threat to external validity.

The paper says, at Table 5

Pith says: minor objection 6 of 6

The table caption says 'Fully correct refinement cases by refinement round.' It may help to add a note that the bold-faced scores indicate the frozen revision used for testing, since this is described only in the text.

What would settle it

Have two independent security researchers (not the original annotators) each re-annotate a random stratified sample of 20 cases (2 per CWE) using the same procedure (compare before/after fix, identify responsible function(s) and statement(s)). Compute percentage agreement and Cohen's kappa for function identification and for statement identification. If agreement falls below 80% (or kappa < 0.7), the reliability of the full annotation set is insufficient and the headline accuracy numbers should be reinterpreted as provisional.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 20 Oct 2026

  1. Reader holds up confidence high
  2. Skeptic agrees 27 s · 21,882 tokens read, 2,412 written
  3. Editor wrote the verdict above 14 s · 22,678 tokens read, 1,098 written
  4. Referee major revision: 2 major objections, 6 minor 30 s · 22,176 tokens read, 2,755 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

This is the first large-scale characterization of how well public security skills actually guide coding agents to find real vulnerabilities. With 37 skills, 4 LLMs, and 14,800 runs on 100 Linux kernel CVEs, the scope is genuinely unprecedented. The headline number—11.1% fully correct—is sobering but credible, and the variation across vulnerability classes and skill workflows is informative.

The experimental design is thorough: controlled runs, multiple models, human-validated failure taxonomy, and evaluation on held-out vulnerabilities. The SKILL-SIF framework is a practical contribution—it shows that skills can be improved by learning from failures, with aggregate fully correct detection rising from 6.05% to 11.20% on unseen cases. The authors also acknowledge their limitations (file-guided setting, single refinement model, ten CWE classes) without hedging into meaningless generalities.

The main soft spot is the ground-truth annotation. The paper does not report inter-annotator reliability for distinguishing vulnerability-relevant operations from supporting changes in fixing patches. This is a standard practice in dataset construction and missing it leaves an open question about reproducibility. However, the authors partially mitigate by allowing multiple valid locations, and the practice of deriving ground truth from patches is widely accepted in the literature. It's a hole, not a fatal flaw.

A minor concern: the refinement uses a single model (gpt-5.6-luna), which raises the possibility that different refinement models would produce different improvements. The authors note this.

Who is this for? Researchers building or evaluating security agents for vulnerability detection. It calibrates expectations and provides a practical improvement direction.

Recommendation: accept after minor revision. Ask the authors to add inter-annotator agreement numbers for the ground-truth annotation and to clarify the annotation instructions. The rest is solid.

The strongest objection

The entire evaluation (Section 4.1) hinges on ground-truth annotations of vulnerable function, statement, and CWE for each of the 100 cases. Section 3.2 describes constructing these by comparing fixing patches with before/after source and distinguishing vulnerability-relevant operations from supporting changes through human judgment. However, the paper does not report how many annotators performed this, what instructions they followed, or what inter-annotator agreement was reached. The paper validates the failure taxonomy with 374 human-annotated samples (80.5% agreement, Section 4.3.1) but does not apply a similar reliability check to the ground truth itself. Without such a check, it is unknown whether the annotations are reproducible. If the ground truth is systematically off—e.g., labeling a refactoring change as the root cause, or missing a multi-step exploit path—the reported 11.1% fully correct rate and all derived comparisons could shift substantially. The paper partially mitigates by allowing multiple valid locations, but the core distinction between vulnerability-relevant and supporting changes remains subjective and unmeasured.

The reader flagged ground-truth annotations as the weakest assumption. I agree that this is the most load-bearing concern because all correctness measurements depend on it. However, I recommend the verdict remain UNCHANGED for two reasons. First, the paper explicitly allows multiple valid functions and statements per vulnerability, which reduces the impact of annotation ambiguity. Second, the paper's central contributions—the relative comparison across skills, the failure taxonomy, and the SKILL-SIF improvement—are unlikely to be invalidated by modest annotation errors; even if the absolute fully correct rate shifts by a few percentage points, the qualitative patterns (e.g., bounds-related vulnerabilities easier than synchronization, high-performing skills use two-stage workflows) remain sound. The concrete test would settle whether the concern quantitatively lands, but the existing evidence and partial mitigations support the reader's ACCEPT verdict.

The referee's report

If the findings hold, this paper makes several important contributions: (1) It provides the first large-scale, controlled evaluation of public security skills for agent-based vulnerability detection, establishing a realistic baseline (11.1% fully correct) that sharply bounds expectations for current coding agents. (2) The failure taxonomy is grounded in human-validated inductive analysis and can inform the design of more robust skills. (3) The SKILL-SIF framework demonstrates that execution feedback can meaningfully improve skill-guided detection on unseen vulnerabilities, with the gains concentrated in reducing skill-level limitations. The study design—controlled runs, multiple models, human validation of the failure taxonomy, separate holdout evaluation for improvement—lends credibility to the empirical results.

Pith's reading of the paper (a machine's summary, not the authors')

The paper investigates how effectively public security skills guide LLM-based coding agents to detect and localize real vulnerabilities. Across 14,800 controlled runs with 37 skills, four underlying LLMs, and 100 real Linux kernel vulnerabilities spanning ten CWE classes, only 11.1% of runs simultaneously identify the correct vulnerability class, the vulnerable function, and the responsible statement. The authors propose SKILL-SIF, an iterative self-improving framework that diagnoses failures from missed vulnerabilities, extracts recurring failure patterns, and mutates the skill procedure accordingly. When applied to the five top-performing skills and tested on 100 previously unseen vulnerabilities, SKILL-SIF improves fully correct detection from 6.05% to 11.20% aggregate (1.85×), with up to 8× improvement for one skill–model combination. The work establishes that security skills provide useful but incomplete guidance and that failure-driven skill evolution can generate transferable improvements.

What carries the argument. The central mechanism is SKILL-SIF, an iterative self-improving framework that converts execution failures into persistent procedural knowledge. Starting from an initial SKILL.md, the framework (1) executes the skill on vulnerable programs, (2) evaluates each run against ground-truth CWE, function, and statement annotations, (3) diagnoses unsuccessful trajectories using a validated LLM-based classifier to identify primary failure mechanisms, (4) aggregates case-level failures into recurring lessons, and (5) mutates the SKILL.md via a refinement model (gpt-5.6-luna). The revised skill is re-executed over the same refinement set, and the best-performing generation is frozen after at most 10 rounds (early termination if no improvement for two consecutive rounds). The framework is applied independently to five top-performing skills, using the original 100 vulnerabilities as the refinement set and a separate 100 unseen vulnerabilities for testing.

Core claim. The paper's central empirical claim is that existing security skills for LLM coding agents offer limited but improvable guidance. In the largest controlled study of its kind (37 skills, 4 LLMs, 100 CVEs, 14,800 runs), only 11.1% of runs achieve full correctness (correct CWE, function, and statement). The breakdown is 24.4% function localization, 18.3% joint function-and-statement localization, and 15.7% class identification. Performance is highly uneven: bounds-related vulnerabilities (CWE-787, CWE-125) are detected at 39.53% and 29.19% fully correct, while synchronization, resource, and initialization classes (e.g., CWE-667, CWE-362, CWE-401) remain below 5%. Failure analysis yields a taxonomy of 14 mechanisms, split between agent execution failures (e.g., relevant evidence not analyzed, 29.3%) and skill-level limitations (e.g., investigation coverage gaps, 15.8%). The paper then demonstrates that failure-driven skill refinement via the SKILL-SIF framework (execute–diagnose–evolve–validate cycle) raises fully correct detection on an independent test set from 6.05% to 11.20% (1.85×), with skill-level limitations nearly eliminated after refinement.

The premise it rests on. The ground-truth annotations are derived from fixing patches by comparing before-fix and after-fix source code, with human judgment distinguishing vulnerability-relevant operations from supporting changes; if the annotations misattribute which operations are responsible, then all subsequent correctness measurements would be affected.

If the paper is right (Pith's inferences, not the authors')

Security skills can be systematically improved from execution feedback without modifying the underlying LLM, offering a practical path to better agent-based vulnerability detection.

Skill designers should incorporate broad candidate discovery followed by a verification stage, and explicitly guide reasoning about ownership, lifetime, synchronization, and error-state relations—which are poorly served by current skills.

Failure-driven refinement largely eliminates skill-level limitations, but agent-execution failures (e.g., missing evidence) persist, indicating that stronger procedures have a ceiling set by the model's own reasoning capabilities.

Improvements from SKILL-SIF transfer across four LLMs (Sonnet-5, Haiku-4.5, DeepSeek-V4-Pro, DeepSeek-V3.2), but the magnitude varies (from +1.0 to +9.8 percentage points), suggesting that procedure robustness remains model-dependent.

The taxonomy of 14 failure mechanisms provides a diagnostic tool for evaluating and improving other security analysis procedures beyond the five skills studied.

The near-elimination of skill-level limitations after refinement suggests that many deficiencies in widely used security skills are correctable without retraining the LLM, lowering the bar for deploying agent-based security audits in practice.

Because the refinement set consists of the same 100 CVEs used in the initial study, the claim of transfer to unseen vulnerabilities would be strengthened by testing on a held-out set drawn from a different time window or kernel subsystem; the authors do this with a second 100-CVE set from the following month.

The sharp early improvement (84% of total gain in the first round) and subsequent stagnation or regression for some skills implies that a single adaptive pass may capture most of the actionable lessons, and that later rounds risk overfitting to the refinement set.

If ground-truth annotations derived from fixing patches occasionally misattribute vulnerability-causing operations (e.g., conflating root cause with cleanup), the reported accuracy numbers could be biased; a complementary annotation by independent security experts on a subset would make the improvement claims more robust.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

Large language model (LLM) coding agents, such as Claude Code, are increasingly used for security auditing, with task-specific skills providing reusable vulnerability-analysis procedures. However, their effectiveness in detecting real vulnerabilities and the reasons for their failures remain unclear. We conduct the first large-scale empirical study of 37 security skills using Claude Code with four underlying LLMs on 100 real Linux kernel vulnerabilities across ten classes, totaling 14800 runs. We evaluate vulnerability detection at the function, statement, and vulnerability-class levels. Results show that 24.4 percent of runs locate the correct function, 18.3 percent identify both the function and responsible statement, and 15.7 percent correctly classify the vulnerability. Our failure analysis reveals limitations in evidence acquisition, state-relation reasoning, code interpretation, candidate verification, investigation coverage, and reporting. Based on these findings, we propose SKILL-SIF, a self-improving framework that iteratively evolves security skills by learning from missed vulnerabilities and failure patterns. We evolve five top-performing skills using the original benchmark and evaluate them on 100 previously unseen vulnerabilities. SKILL-SIF improves fully correct vulnerability detection by up to eight times (from 2 percent to 16 percent) on unseen vulnerabilities, while aggregate accuracy across four LLMs increases by 1.85 times (from 6.05 percent to 11.20 percent). These findings highlight the limitations of existing security skills and demonstrate the potential of failure-driven skill evolution to improve vulnerability detection in coding agents.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-10-20

The authors' main claim, as the classifier picked it out 1.00

“These findings highlight the limitations of existing security skills and demonstrate the potential of failure-driven skill evolution to improve vulnerability detection in coding agents.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.94 · states limitation 0.47 · human participants 0.04 · controlled comparison 0.74 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.78 · mentions code or data availability 0.04

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Large language model (LLM) coding agents, such as Claude Code, are increasingly used for security auditing, with task-specific skills providing reusable vulnerability-analysis procedures. background 1.00 0.04 0.02 asserted 0.99
However, their effectiveness in detecting real vulnerabilities and the reasons for their failures remain unclear. gap 1.00 0.05 0.02 asserted 0.89
We conduct the first large-scale empirical study of 37 security skills using Claude Code with four underlying LLMs on 100 real Linux kernel vulnerabilities across ten classes, totaling 14800 runs. method 0.52 0.97 0.03 asserted 0.99
We evaluate vulnerability detection at the function, statement, and vulnerability-class levels. method 0.97 0.90 0.02 asserted 0.95
Results show that 24.4 percent of runs locate the correct function, 18.3 percent identify both the function and responsible statement, and 15.7 percent correctly classify the vulnerability. result 1.00 0.97 0.10 asserted 1.00
Our failure analysis reveals limitations in evidence acquisition, state-relation reasoning, code interpretation, candidate verification, investigation coverage, and reporting. result 0.73 0.95 0.10 asserted 1.00
Based on these findings, we propose SKILL-SIF, a self-improving framework that iteratively evolves security skills by learning from missed vulnerabilities and failure patterns. contribution 0.78 0.81 0.16 asserted 0.97
We evolve five top-performing skills using the original benchmark and evaluate them on 100 previously unseen vulnerabilities. method 0.81 0.97 0.05 asserted 0.93
SKILL-SIF improves fully correct vulnerability detection by up to eight times (from 2 percent to 16 percent) on unseen vulnerabilities, while aggregate accuracy across four LLMs increases by 1.85 times (from 6.05 percent to 11.20 percent). result 1.00 0.97 0.28 asserted 1.00
These findings highlight the limitations of existing security skills and demonstrate the potential of failure-driven skill evolution to improve vulnerability detection in coding agents. interpretation 1.00 0.83 0.87 asserted 1.00

What kind of paper the classifier reports this is: empirical study 0.98 · evidence from observation or experiment 0.99

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-20: 1 request, 7,646 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (8)
Figure 1 of the paper
Figure 1. Empirical study overview unsafe tool use. Together, these studies establish skills as an important adaptation layer for agent behavior. Our study focuses on their effectiveness for security analysis, evaluating whether public vulnerability-detection skills can guide coding agents to correctly identify the vulnerability class and localize the responsible function and statement in real Linux kernel revisions. 2.3 Skill Refinement and Evolution Recent work has also explored how agent skills can be improved from execution experience. SkillRevise [17] diagnoses defects from execution…
Figure 2 of the paper
Figure 2. Model performance and vulnerability coverage. Fully correct, partial, and incorrect results are reported
Figure 3 of the paper
Figure 3. Performance across localization and vulnerability identification. For each model, the bars show the
Figure 4 of the paper
Figure 4. Vulnerability detection and localization performance across CWE classes. Classes are ordered by fully
Figure 5 of the paper
Figure 5. Fully correct rates of the 37 evaluated skills, ranked from highest to lowest. Each bar shows the
Figure 6 of the paper
Figure 6. Examples of agent-execution and skill-level failure mechanisms.
Figure 7 of the paper
Figure 7. Iterative Skill Self-Improving Framework
Figure 8 of the paper
Figure 8. Change in the distribution of failure mechanisms from baseline to refined skills on the held-out test

Figures are the authors', reproduced from the paper.

References (51)
  1. [1] Anthropic. 2025. Claude Haiku 4.5 System Card. https://www-cdn.anthropic.com/ 7aad69bf12627d42234e01ee7c36305dc2f6a970.pdf. Accessed: Aug. 31, 2026 on Pith
  2. [2] Anthropic. 2026. Anthropic Transparency Hub. https://www.anthropic.com/transparency. Accessed: Aug. 31, 2026. , Vol. 1, No. 1, Article . Publication date: October 2026. How Effectively Do Security Skills Guide Coding Agents in Vulnerability Detection in Linux Kernel? 19 on Pith
  3. [3] Pavel Avgustinov, Oege de Moor, Michael Peyton Jones, and Max Schäfer. 2016. QL: Object-Oriented Queries on Relational Data. In30th European Conference on Object-Oriented Programming (ECOOP 2016). 2:1–2:25. doi:10.4230/ LIPIcs.ECOOP.2016.2 on Pith
  4. [4] Mingi Cho, Dohyeon An, Hoyong Jin, and Taekyoung Kwon. 2023. BoKASAN: Binary-only Kernel Address Sanitizer for Effective Kernel Fuzzing. InUSENIX Security 23. USENIX Association, Anaheim, CA, 4985–5002. https://www. usenix.org/conference/usenixsecurity23/presentation/cho on Pith
  5. [5] DeepSeek. 2025. DeepSeek-V3.2 Release. https://api-docs.deepseek.com/news/news251201/. Accessed: Aug. 31, 2026 on Pith
  6. [6] DeepSeek. 2026. DeepSeek API Updates. https://api-docs.deepseek.com/updates/. Accessed: Aug. 31, 2026 on Pith
  7. [7] Yangruibo Ding, Yanjun Fu, Omkar Chen, Chace Fuller, Hao Jiang, Yiling Lou, Xinyu Zhang, and Michael R. Lyu on Pith
  8. [8] Alex Liu et al. 2026. Human-LLM Collaborative Inductive Coding for Conceptualizing K-12 Educator AI Use. arXiv:2607.28889 [cs.HC] https://arxiv.org/abs/2607.28889 on PithPith's reviewarXiv
  9. [9] Hamed Jelodar et al. 2026. Large language model (LLM) for software security: Code analysis, malware analysis, reverse engineering.Journal of Information Security and Applications98 (2026), 104390. doi:10.1016/j.jisa.2026.104390 on Pithdoi:10.1016/j.jisa.2026.104390
  10. [10] Hanrong Zhang et al. 2026. CoEvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification. arXiv:2604.01687 [cs.AI] https://arxiv.org/abs/2604.01687 on PithPith's reviewarXiv
  11. [11] Insu Yun et al. 2018. QSYM : A Practical Concolic Execution Engine Tailored for Hybrid Fuzzing. InUSENIX Security 18. USENIX Association, Baltimore, MD, 745–761. https://www.usenix.org/conference/usenixsecurity18/presentation/yun on Pith
  12. [12] Junjie Wang et al. 2026. A Survey for LLM Agent Trajectory Analysis: From Failure Attribution to Enhancement.IEEE Transactions on Software Engineering(2026). Publisher Copyright:©1976-2012 IEEE.. doi:10.1109/TSE.2026.3717765 on Pithdoi:10.1109/tse.2026.3717765
  13. [13] Lakshya A Agrawal et al. 2026. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457 [cs.CL] https://arxiv.org/abs/2507.19457 on PithPith's reviewarXiv
  14. [14] Xiangyi Li et al. 2026. SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks. arXiv:2602.12670 [cs.AI] https://arxiv.org/abs/2602.12670 on PithPith's reviewarXiv
  15. [15] Xiangxin Zhao et al. 2026. Failure as a Process: An Anatomy of CLI Coding Agent Trajectories. arXiv:2607.09510 [cs.SE] https://arxiv.org/abs/2607.09510 on PithPith's reviewarXiv
  16. [16] Yuxuan Liu et al. 2026. Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds. arXiv:2608.02636 [cs.SE] https://arxiv.org/abs/2608.02636 on PithPith's reviewarXiv
  17. [17] Yuxuan Liu et al. 2026. SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision. arXiv:2606.01139 [cs.AI] https://arxiv.org/abs/2606.01139 on PithPith's reviewarXiv
  18. [18] Zhiyuan Jiang et al. 2026. Demystifying Agent Skills: Why They Work-Until They Don’t. arXiv:2608.14036 [cs.AI] https://arxiv.org/abs/2608.14036 on PithPith's reviewarXiv
  19. [19] Google. 2026. syzkaller: An Unsupervised Coverage-Guided Kernel Fuzzer. https://github.com/google/syzkaller. Accessed 2026 on Pith
  20. [20] Kyungtae Kim, Dae Jeong, Chung Hwan Kim, Yeongjin Jang, Insik Shin, and Byoungyoung Lee. 2020. HFL: Hybrid Fuzzing on the Linux Kernel. doi:10.14722/ndss.2020.24018 on Pithdoi:10.14722/ndss.2020.24018
show 31 more
  1. [21] Greg Kroah-Hartman. 2007. Linux Kernel Development: How Fast it is Going, Who is Doing It, What They are Doing, and Who is Sponsoring It. InProceedings of the Linux Symposium, Vol. 1. Ottawa, Ontario, Canada, 239–244. https://www.kernel.org/doc/ols/2007/ols2007v1-pages-239-244.pdf on Pith
  2. [22] Elias et al. Kuiter. 2025. How Configurable Is the Linux Kernel? Analyzing Two Decades of Feature-Model History. ACM Trans. Softw. Eng. Methodol.35, 1, Article 27 (Dec. 2025), 48 pages. doi:10.1145/3729423 on Pithdoi:10.1145/3729423
  3. [23] Julia Lawall and Gilles Muller. 2018. Coccinelle: 10 Years of Automated Evolution in the Linux Kernel. In2018 USENIX Annual Technical Conference (USENIX ATC 18). USENIX Association, Boston, MA, 601–614. https://www.usenix.org/ conference/atc18/presentation/lawall on Pith
  4. [24] Ziyang Li, Saikat Dutta, and Mayur Naik. 2025. IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities. InInternational Conference on Learning Representations, Y. Yue, A. Garg, N. Peng, F. Sha, and R. Yu (Eds.), Vol. 2025. 35735– 35758. https://proceedings.ic... on Pith
  5. [25] Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018. VulDeePecker: A Deep Learning-Based System for Vulnerability Detection. InNDSS 2018. Internet Society. doi:10.14722/ndss.2018.23158 on Pithdoi:10.14722/ndss.2018.23158
  6. [26] Zhen et al. Li. 2022. SySeVR: A Framework for Using Deep Learning to Detect Software Vulnerabilities.IEEE Transactions on Dependable and Secure Computing19, 4 (July 2022), 2244–2258. doi:10.1109/tdsc.2021.3051525 on Pithdoi:10.1109/tdsc.2021.3051525
  7. [27] Jie Lin and David Mohaisen. 2025. From Large to Mammoth: A Comparative Evaluation of Large Language Models in Vulnerability Detection. InProceedings of the Network and Distributed System Security Symposium (NDSS). https: //www.ndss-symposium.org/wp-content/uploads/2025-1491-pa... on Pith
  8. [28] George Ling, Shanshan Zhong, and Richard Huang. 2026. Agent Skills: A Data-Driven Analysis of Claude Skills for Extending Large Language Model Functionality. arXiv:2602.08004 [cs.SE] https://arxiv.org/abs/2602.08004 on PitharXiv
  9. [29] Stephan Lipp, Sebastian Banescu, and Alexander Pretschner. 2022. An empirical study on the effectiveness of static C code analyzers for vulnerability detection. InISSTA 2022(Virtual, South Korea). Association for Computing Machinery, New York, NY, USA, 544–555. doi:10.1145/353... on Pithdoi:10.1145/3533767.3534380
  10. [30] Yujian Liu, Jiabao Ji, Li An, Tommi Jaakkola, Yang Zhang, and Shiyu Chang. 2026. How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings. arXiv:2604.04323 [cs.CL] https://arxiv.org/abs/2604.04323 on PithPith's reviewarXiv
  11. [31] Huy Nguyen and Anh Nguyen. 2025. Enhancing Static Analysis with LLMs to Detect Software Vulnerabilities. (2025). Preprint. https://escholarship.org/content/qt0kj3k9h9/qt0kj3k9h9.pdf on Pith
  12. [32] Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xi- aoxi Jiang, and Guanjun Jiang. 2026. Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills. arXiv:2603.25158 [cs.AI] https://arxiv.org/abs/2603.25158 on PithPith's reviewarXiv
  13. [33] Shankara Pailoor, Andrew Aday, and Suman Jana. 2018. MoonShine: Optimizing OS Fuzzer Seed Selection with Trace Distillation. In27th USENIX Security Symposium (USENIX Security 18). USENIX Association, Baltimore, MD, 729–743. https://www.usenix.org/conference/usenixsecurity18/pr... on Pith
  14. [34] Hammond Pearce, Benjamin Tan, Baleegh Ahmad, Ramesh Karri, and Brendan Dolan-Gavitt. 2023. Examining Zero- Shot Vulnerability Repair with Large Language Models. InProceedings of the 2023 IEEE Symposium on Security and Privacy. doi:10.1109/SP46215.2023.10179324 on Pithdoi:10.1109/sp46215.2023.10179324
  15. [35] Sebastian Poeplau and Aurélien Francillon. 2020. Symbolic execution with SymCC: Don’t interpret, compile!. InUSENIX Security 20. USENIX Association, 181–198. https://www.usenix.org/conference/usenixsecurity20/presentation/poeplau on Pith
  16. [36] David Schmotz, Sahar Abdelnabi, and Maksym Andriushchenko. 2025. Agent Skills Enable a New Class of Realistic and Trivially Simple Prompt Injections. arXiv:2510.26328 [cs.LG] https://arxiv.org/abs/2510.26328 on PitharXiv
  17. [37] David Schmotz, Luca Beurer-Kellner, Sahar Abdelnabi, and Maksym Andriushchenko. 2026. Skill-Inject: Measuring Agent Vulnerability to Skill File Attacks. arXiv:2602.20156 [cs.CR] https://arxiv.org/abs/2602.20156 on PithPith's reviewarXiv
  18. [38] Ze et al. Sheng. 2025. LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights.ACM Comput. Surv.58, 5, Article 134 (Nov. 2025), 35 pages. doi:10.1145/3769082 on Pithdoi:10.1145/3769082
  19. [39] Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringhini. 2024. LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks. InProceedings of the 2024 IEEE Symposium on... on Pithdoi:10.1109/sp54263.2024.00210
  20. [40] Daimeng Wang, Zheng Zhang, Hang Zhang, Zhiyun Qian, Srikanth V. Krishnamurthy, and Nael Abu-Ghazaleh. 2021. SyzVegas: Beating Kernel Fuzzing Odds with Reinforcement Learning. In30th USENIX Security Symposium (USENIX Security 21). USENIX Association, 2741–2758. https://www.usen... on Pith
  21. [41] Yong Wang et al. 2026. CQLLM: A Framework for Generating CodeQL Security Vulnerability Detection Rules Using Large Language Models.Applied Sciences16, 1 (2026), 517. doi:10.3390/app16010517 on Pithdoi:10.3390/app16010517
  22. [42] Zora Zhiruo Wang, Apurva Gandhi, Graham Neubig, and Daniel Fried. 2025. Inducing Programmatic Skills for Agentic Tasks. arXiv:2504.06821 [cs.CL] https://arxiv.org/abs/2504.06821 on PithPith's reviewarXiv
  23. [43] Wenpeng Xing, Minghao Li, Mohan Li, and Meng Han. 2026. Towards Robust and Secure Embodied AI: A Survey on Vulnerabilities and Attacks.ACM Comput. Surv.58, 12, Article 312 (May 2026), 36 pages. doi:10.1145/3806048 on Pithdoi:10.1145/3806048
  24. [44] Renjun Xu and Yang Yan. 2026. Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward. arXiv:2602.12430 [cs.MA] https://arxiv.org/abs/2602.12430 on PithPith's reviewarXiv
  25. [45] Xiaofang Yang, Ziqi Miao, Dianbo Sui, Jing Shao, and Lijun Li. 2026. Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents. arXiv:2609.01487 [cs.CR] https://arxiv.org/abs/2609.01487 on PithPith's reviewarXiv
  26. [46] Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, Kai Qiu, Yuqing Yang, Dongdong Chen, Xue Yang, and Chong Luo. 2026. SkillOpt: Executive Strategy for Self-Evolving Agent Skills. arXiv:2605.23904 [cs.AI] https://ar... on PithPith's reviewarXiv
  27. [47] Zhuoyun Yu, Xin Xie, Wuguannan Yao, Chenxi Wang, Lei Liang, Xiang Qi, and Shumin Deng. 2026. SkillAdaptor: Self-Adapting Skills for LLM Agents from Trajectories. arXiv:2606.01311 [cs.CL] https://arxiv.org/abs/2606.01311 on PithPith's reviewarXiv
  28. [48] Haozhen Zhang, Quanyu Long, Jianzhu Bao, Tao Feng, Weizhi Zhang, Haodong Yue, and Wenya Wang. 2026. MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agents. arXiv:2602.02474 [cs.CL] https://arxiv.org/abs/2602. 02474 on PithPith's reviewarXiv
  29. [49] Zhibo Zhang, Zhen Ouyang, Ling Shi, and Kailong Wang. 2026. TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation. arXiv:2608.17588 [cs.AI] https://arxiv.org/abs/2608.17588 on PithPith's reviewarXiv
  30. [50] Xin Zhou, Sicong Cao, Xiaobing Sun, and David Lo. 2025. Large Language Model for Vulnerability Detection and Repair: Literature Review and the Road Ahead.ACM Trans. Softw. Eng. Methodol.34, 5, Article 145 (May 2025), 31 pages. doi:10.1145/3708522 , Vol. 1, No. 1, Article . Pub... on Pithdoi:10.1145/3708522
  31. [2025] Vulnerability Detection with Code Language Models: How Far Are We?. InProceedings of the 47th IEEE/ACM International Conference on Software Engineering (ICSE). doi:10.1109/ICSE55347.2025.00038 on Pithdoi:10.1109/icse55347.2025.00038

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment