Artificial Intelligence · arXiv:2608.09253 v1 · submitted 10 Aug 2026 · 42 references
SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance
The paper says
“LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably.”
more of the abstract
Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 11 Aug 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A solid runtime-assurance system with a credible 24.1% success-rate gain, held back mainly by unvalidated LLM extraction and missing artifacts. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 42 references say.
The paper says, at III-B, III-C, III-D
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The runtime assurance mechanism treats the LLM-extracted logical actions and failure patterns as ground truth, but the validation described in Sections III-B and III-C is only structural (unique step IDs, acyclic dependencies, valid regular expressions, well-formed fields). The FSM in Section III-D marks step si complete only when all extracted logical actions are matched, and the procedure checker denies an …
more of objection 1
… action when it matches a failure pattern; if the GPT-5.4 parser or miner hallucinates a logical action, a step can be marked complete before the required operation occurs, and if a necessary logical action is omitted, a correct execution can be denied termination or spuriously re-planned. The paper reports no assessment of whether extracted logical actions are necessary or sufficient for each step, whether failure patterns correspond to actual failure causes, or whether suggestions and warnings contradict the skill document. Because the reported 24.1% improvement is produced through these interventions, the headline result is conditional on the semantic quality of the extracted guidance, which is never measured.
The paper says, at IV-B, Table II
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The evaluation in Table II measures only end-to-end task success rates and their standard deviations; it does not report any analysis of the correctness of the checker's individual decisions. Without data on the number of interventions, the rate of false-positive failure-pattern triggers, whether denials were followed by successful re-planning, and whether termination denials occurred on executions that would have succeeded without intervention, the causal contribution of the proposed runtime assurance mechanism is not established.
more of objection 2
The single case study in Figure 6 is illustrative but does not substitute for a systematic analysis of the checker's precision and recall.
The paper says, at IV-A
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The executability filter in Section IV-A retains only the 15 skills that each of the four agent-model configurations can complete at least once in five executions, and the paper does not report how many skills were excluded or analyze the sensitivity of the headline improvement to this choice.
more of objection 3
This selection biases the evaluation toward skills on which agents already show partial competence, so the 24.1% average improvement should be stated as conditional on this subpopulation rather than as a general property of skill execution. The paper should either report results on the full set with appropriate controls or explicitly bound the claims to the filtered set and discuss the direction of the potential bias.
6 minor objections
The paper says, at III-C, IV-A
Pith says: minor objection 1 of 6
The origin of the historical successful and failed traces used to seed the initial runtime guidance is not specified; the paper should state whether these are the five filtering runs from Section IV-A or a separate collection.
The paper says, at IV-D, Figure 8
Pith says: minor objection 2 of 6
The self-evolving analysis in Figure 8 is shown for a single skill and configuration; a curve averaged over all 15 skills, with confidence bands, should be included in the main text to support the general claim of self-evolution.
show 4 more
The paper says, at Table III
Pith says: minor objection 3 of 6
The 'Token Cost' column in Table III should state the unit explicitly; the values appear to be token counts, but this is not stated in the caption or the table header.
The paper says, at V
Pith says: minor objection 4 of 6
The paired t-test statement in Section V lacks important details: the pairing unit, the number of pairs, the test statistic, and whether any multiple-comparison correction was applied. Please provide this information or release the per-run results needed to verify it.
The paper says, at III-B, III-C, IV-A
Pith says: minor objection 5 of 6
Since GPT-5.4 is used both as the parser/miner and as one of the backbone models (Codex with GPT-5.4), the paper should discuss any potential shared-model confound, even if the evaluation queries are held out, to reassure readers that the extracted guidance is not implicitly tuned to the same model's stylistic preferences.
The paper says, at VI
Pith says: minor objection 6 of 6
The claim of being 'the first' runtime assurance framework for skill-oriented agent execution is a strong novelty statement; consider softening it or providing a more systematic search of the literature to support the claim.
What would settle it
Release the 15 runtime-guidance instances and have two independent annotators label every logical action, failure pattern, suggestion, and warning as correct, incorrect, or incomplete against the skill document and a sample of evolution traces. Then rebuild SkillSentry with the corrected guidance, removing or fixing all incorrect or incomplete items, and rerun the held-out Qtest evaluations. Pre-specify a threshold: if the corrected average success rate is more than 5 percentage points below the reported 77.7%, semantic extraction quality is load-bearing; if it matches, the assurance mechanism itself, rather than the specific mined content, drives the improvement and the concern does not land.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 11 Aug 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Read SkillSentry. What is new here is the combination: a DSL that holds both the parsed skill specification and mined execution experience, wrapped around the agent loop with step-aware checking. That is a genuinely useful design, and the evaluation is more careful than most: 15 skills, four agent-model configurations, held-out task instances, five repeated runs, ablations, and a self-evolution curve. The headline claim — average success from 62.6% to 77.7%, with standard deviation down 41.1% — looks internally consistent. The per-skill table shows improvement in every one of 60 configurations, and the paired t-test is reported. The overhead numbers are believable: 7.8% more turns, 0.8% of execution time in its own computation.
Credit where due: the ablation isolates on-enter guidance, failure patterns, and step-level delivery, and the system-prompt comparison is a fair control. The Fig. 6 case study is illustrative but not load-bearing.
Soft spots, in order of size. First, semantic extraction is load-bearing and only structurally validated. The GPT-5.4 parser and miner decide what counts as a logical action, a failure pattern, and a useful suggestion; validation checks unique step IDs, acyclic dependencies, and valid regexes. A hallucinated logical action can mark a step complete before the required operation happens, or a missing one can deny termination on a correct execution. Both change the interventions behind the 24.1% gain. No parser/miner output or guidance instance is released, so this cannot be checked. Second, the skill filter keeps only 15 skills where all four agent-model pairs succeeded at least once in five runs. That is a sensible way to study instability rather than capability, but it softens the general benchmark claim. Third, there is no direct comparison with AgentSpec, ShieldAgent, or similar runtime enforcement. The paper justifies the omission, but the reader cannot tell how much of the gain is skill-specific versus generic guardrailing. Fourth, minor: GPT-5.4 generates paraphrases, mines traces, and also serves as a backbone; the held-out split keeps this from explaining the results, but it is a slight self-reference.
Who this is for: people building runtime layers for LLM agents and researchers measuring skill reliability. It deserves a serious referee. I would ask for artifact release, semantic spot-checks of extracted guidance, and at least one direct baseline before treating 24.1% as a settled benchmark. The architecture and main effect are credible.
The strongest objection
The most load-bearing link in the argument is the unvalidated semantic correctness of the LLM-generated runtime guidance. In Sec. III-B, a GPT-5.4 parser converts each natural-language skill document into DSL steps, dependencies, constraints, and termination requirements; in Sec. III-C, a GPT-5.4 miner converts successful/failed traces into logical actions, suggestions, warnings, and failure patterns. The runtime checker in Sec. III-D then builds an FSM in which a step is completed only when all extracted logical actions match, and it denies or replans on failure patterns. The paper's validation is structural only: unique step IDs, dependency acyclicity, valid regular expressions, and well-formed fields. Nothing checks whether a parsed logical action is actually necessary or sufficient for the step, whether a failure pattern truly reflects a failure cause, or whether a suggestion contradicts the skill document. A hallucinated logical action can mark a step complete before the required operation occurs; a missing logical action can deny termination on a correct execution and trigger spurious replanning. Both failure modes change the interventions that produce the reported 24.1% gain. Since no parser/miner output or artifact is released, the headline result cannot be separated from extraction quality.
The reader identified the same concern, and the stress test confirms it is the correct soft spot. The architecture is coherent and the empirical evaluation is internally consistent: recomputing Table II row averages reproduces the aggregate 62.6% to 77.7% transition, a relative improvement of 24.1%. The per-skill positive results are extensive, and the ablation supports the claimed mechanism. The missing piece is not arithmetic; it is an unverified semantic layer on which the runtime FSM depends. Structural validation cannot catch a false-positive or false-negative logical action, and both would change the interventions that produce the measured gains. A human-annotation plus corrected-guidance rerun would distinguish 'runtime assurance works because the extracted guidance is semantically right' from 'runtime assurance works even with imperfect extraction.' Because this is an addressable empirical gap rather than a demonstrated error, the reader's CONDITIONAL verdict remains appropriate; artifact release and a semantic audit should be conditions. I would not move to REJECT.
The referee's report
If the claimed effects hold, SkillSentry would demonstrate that runtime assurance, rather than model retraining or prompt rewriting, can substantially improve the reliability of skill execution in LLM agents. The paper has notable strengths: a broad evaluation covering 60 skill-model combinations, held-out test queries with task-level splits, five repeated runs per condition, ablations isolating components, a transfer study, and a fail-open design that reduces risk of disruption. The use of deterministic verifiers for trace labeling is also a positive. However, the central claim is conditional on the semantic correctness of LLM-generated runtime guidance, which is not validated beyond structural checks; the paper also lacks intervention-level analysis that would show the checker's decisions themselves are reliable. These gaps make the current evidence suggestive rather than conclusive, though the outcome is important enough to warrant further work.
Pith's reading of the paper (a machine's summary, not the authors')
SkillSentry claims that many residual failures of LLM agents are not capability failures but execution-instability failures, and that a runtime assurance wrapper can catch and correct them without retraining or rewriting prompts. It builds structured runtime guidance for each skill by parsing the skill document and mining historical successful and failed traces, then wraps the agent loop with a procedure checker and a termination checker that nudge the agent back on procedure. Across 15 skills, two agent runtimes, and four backbone models, the paper reports average task success rising from 62.6% to 77.7%, a relative gain of 24.1%, while the standard deviation across repeated runs falls by 41.1% and the checker itself uses about 0.8% of total execution time. If that holds, runtime assurance becomes a practical complement to model training and prompt engineering for making LLM agents dependable.
What carries the argument. The load-bearing object is the runtime guidance DSL: each skill becomes a list of steps, each with a unique identifier, dependency list, tool and parameter constraints, logical actions with regular-expression action patterns, on-enter suggestions and warnings, and failure patterns with reasons, plus a termination list marking steps that must be completed. An LLM-based parser extracts the specification from the skill document, an LLM-based miner fills the experience fields from labelled traces, and at runtime the guidance is compiled into a finite-state machine whose states are skill steps and whose completion condition is that all logical actions of a step have been matched. The procedure checker advances this FSM, delivers step guidance, and temporarily denies actions that violate dependencies or match failure patterns, while the termination checker blocks final output until the required steps are done.
Core claim. The paper's central claim is that unreliable skill execution is largely procedural: agents skip required steps, execute steps out of order, or use wrong parameters, even when they have demonstrated the capability to finish the task before. SkillSentry encodes what a skill is supposed to do in a small domain-specific language, seeds that encoding from the skill document as a specification and from successful and failed traces as execution experience, and then enforces it live. It matches each planned action against expected logical actions, temporarily denies actions that skip dependencies or repeat historically failing patterns, injects step-level suggestions and warnings at the right moment, and refuses termination until all required steps are complete. The evaluation claims this raises average success from 62.6% to 77.7% and cuts repeated-run standard deviation by 41.1%, with gains on every tested configuration and the largest relative gains on skills with the weakest baselines.
The premise it rests on. The framework's entire benefit rests on the language model that reads skill documents and execution traces turning them into accurate runtime guidance; the paper checks only formatting-level properties, not whether the extracted steps, constraints, and failure patterns actually mean what the skill requires.
If the paper is right (Pith's inferences, not the authors')
If the gains hold, an agent that already knows a skill can be made reliable by wrapping rather than retraining: the same capability, once verified, becomes reproducible across similar queries and repeated runs.
Because the checker is fail-open and adds only about 0.8% of execution time, it can sit in production agent loops without becoming the bottleneck.
Step-level guidance delivery beats putting all guidance in the system prompt by 5.8%, implying that presenting guidance in the right execution context matters as much as the guidance content.
Runtime guidance evolved with one backbone model retains roughly 94.2% of its native benefit when transferred to another backbone under the same agent, so assurance artifacts are partially portable.
Self-evolution over ten iterations delivers most of its benefit early, suggesting the guidance converges quickly and a modest trace collection budget suffices.
The paper evaluates only skills on which each agent succeeded at least once, so the method is aimed at instability rather than capability; extending it to never-succeeded skills would test whether its experience mining can bootstrap without positive traces.
Because the parser and miner are themselves LLM calls, the framework's ceiling likely tracks the extraction model's fidelity; a cheaper model might reproduce the results only if the extracted guidance stays semantically correct.
SkillSentry is complementary to offline prompt optimization: it corrects transient deviations at runtime, while prompt rewriting changes the distribution those deviations arise from, so combining both could push reliability higher than either alone.
The termination checker only knows the required steps the parser extracted; adding a check on the final output's semantic correctness would catch cases where all steps ran but the result is still wrong.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.11 · human participants 0.04 · controlled comparison 0.35 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.33 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. | background 1.00 | 0.05 | 0.02 | asserted 0.99 |
| Although skills provide reusable procedural knowledge, agents may still execute them unreliably. | background 0.77 | 0.07 | 0.02 | hedged possibility 0.99 |
| Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. | background 0.94 | 0.10 | 0.03 | hedged possibility 0.97 |
| Such instability limits the practical reliability of LLM agents. | background 0.91 | 0.06 | 0.03 | asserted 1.00 |
| To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. | contribution 0.96 | 0.86 | 0.07 | asserted 0.99 |
| SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. | method 0.99 | 0.87 | 0.03 | asserted 1.00 |
| It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. | method 1.00 | 0.78 | 0.03 | asserted 0.99 |
| We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. | method 0.95 | 0.95 | 0.03 | asserted 0.86 |
| Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs. | result 1.00 | 0.97 | 0.90 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,975 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (9)
Showing 6 of the paper's 9 figures. Figures are the authors', reproduced from the paper.
References (42)
- [1] Anonymous GitHub. (2026) SkillSentry. Accessed on May 11, 2026. [Online]. Available: https://anonymous.4open.science/r/skillsentry-865C on Pith
- [2] Anthropic, “Introducing claude haiku 4.5,” https://www.anthropic.com/ news/claude-haiku-4-5, Oct. 2025, accessed on June 8, 2026 on Pith
- [3] ——. (2025) Writing effective tools for agents — with agents. Accessed on May 13, 2026. [Online]. Available: https://www.anthropic.com/engi neering/writing-tools-for-agents on Pith
- [4] ——, “Claude code,” https://code.claude.com/docs/en/overview, 2026, accessed on June 8, 2026 on Pith
- [5] ——, “Claude code docs: Hooks reference,” https://code.claude.com/do cs/en/hooks, 2026, accessed on June 13, 2026 on Pith
- [6] ——, “Introducing claude opus 4.6,” https://www.anthropic.com/news/c laude-opus-4-6, Feb. 2026, accessed on June 8, 2026 on Pith
- [7] Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnonet al., “Constitutional ai: Harmlessness from ai feedback,”arXiv preprint arXiv:2212.08073, 2022 on PithPith's reviewarXiv
- [8] I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, 2025, pp. 2188– 2200 on Pith
- [9] I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,” inProceedings of the 2025 40th IEEE/ACM International Conference on Automated Software Engineering, 2025, pp. 2846–2857 on Pith
- [10] J. F. Box, “Guinness, gosset, fisher, and small samples,”Statistical Science, pp. 45–52, 1987 on Pith
- [11] Z. Chen, M. Kang, and B. Li, “Shieldagent: Shielding agents via verifiable safety policy reasoning,” inProceedings of the International Conference on Machine Learning, 2025, pp. 8313–8344 on Pith
- [12] Google. (2023) Google Gemini. Accessed on May 13, 2026. [Online]. Available: https://gemini.google.com/ on Pith
- [13] A. Gupta, “Reliabilitybench: Evaluating llm agent reliability under production-like stress conditions,”arXiv preprint arXiv:2601.06112, 2026 on PitharXiv
- [14] X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024 on Pith
- [15] T. Kumar, A. Tripathy, G. Saranathan, M. Foltin, S. Bhattacharya, S. Hinchley, D. M. Bahls, D. Brookshire, L. Kaplan, and R. W. Wisniewski, “Infrastructuresentinel: Policy enforced guardrails for secure mcp-driven infrastructure agents,” inProceedings of the AAAI Conference on... on Pith
- [16] X. Li, W. Chen, Y . Liu, S. Zheng, X. Chen, Y . He, Y . Li, B. You, H. Shen, J. Sunet al., “Skillsbench: Benchmarking how well agent skills work across diverse tasks,”arXiv preprint arXiv:2602.12670, 2026 on PithPith's reviewarXiv
- [17] Y . Li, H. Wen, Y . Chen, H. Liu, Y . Tian, and Y . Feng, “No attack required: Semantic fuzzing for specification violations in agent skills,” arXiv preprint arXiv:2605.13044, 2026 on PithPith's reviewarXiv
- [18] J. Ni, Y . Liu, X. Liu, Y . Sun, M. Zhou, P. Cheng, D. Wang, E. Zhao, X. Jiang, and G. Jiang, “Trace2skill: Distill trajectory-local lessons into transferable agent skills,”arXiv preprint arXiv:2603.25158, 2026 on PithPith's reviewarXiv
- [19] OpenAI. (2023) Introducing ChatGPT. Accessed on May 13, 2026. [Online]. Available: https://openai.com/blog/chatgpt on Pith
- [20] OpenAI, “Introducing gpt-5.2,” https://openai.com/index/introducing-gpt -5-2/, Dec. 2025, accessed on June 8, 2026 on Pith
show 22 more
- [21] ——, “Codex,” https://developers.openai.com/codex/, 2026, accessed on June 8, 2026 on Pith
- [22] ——, “Codex docs: Hooks,” https://developers.openai.com/codex/hooks, 2026, accessed on June 13, 2026 on Pith
- [23] ——, “Introducing gpt-5.4,” https://openai.com/index/introducing-gpt-5 -4/, Mar. 2026, accessed on June 8, 2026 on Pith
- [24] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022 on Pith
- [25] OWASP Foundation, “Improper error handling,” https://owasp.org/ww w-community/Improper Error Handling, 2026, accessed on June 25, 2026 on Pith
- [26] R. Pryzant, D. Iter, J. Li, Y . T. Lee, C. Zhu, and M. Zeng, “Automatic prompt optimization with “gradient descent” and beam search,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 7957–7968 on Pith
- [27] Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qianet al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” inProceedings of the International Conference on Learning Representations, vol. 2024, 2024, pp. 9695–9717 on Pith
- [28] T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in Neural Informa- tion Processing Systems, vol. 36, pp. 68 539–68 551, 2023 on Pith
- [29] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: language agents with verbal reinforcement learning,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023, pp. 8634–8652 on Pith
- [30] G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”arXiv preprint arXiv:2305.16291, 2023 on PithPith's reviewarXiv
- [31] H. Wang, C. M. Poskitt, and J. Sun, “Agentspec: Customizable runtime enforcement for safe and reliable LLM agents,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering, 2026 on Pith
- [32] Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liuet al., “Autogen: Enabling next-gen llm applications via multi-agent conversations,” inProceedings of the 1st Conference on Language Modeling, 2024 on Pith
- [33] Z. Xue, Y . Zhao, S. Wang, K. Chen, and H. Wang, “A characterization study of bugs in llm agent workflow orchestration frameworks,” in Proceedings of the 2025 40th IEEE/ACM International Conference on Automated Software Engineering, 2025, pp. 3369–3380 on Pith
- [34] C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inProceedings of the 12th International Conference on Learning Representations, 2024 on Pith
- [35] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024 on Pith
- [36] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022 on PithPith's reviewarXiv
- [37] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xieet al., “Glm-5: from vibe coding to agentic engineering,” arXiv preprint arXiv:2602.15763, 2026 on PithPith's reviewarXiv
- [38] Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604 on Pith
- [39] A. Zhao, D. Huang, Q. Xu, M. Lin, Y .-J. Liu, and G. Huang, “Expel: Llm agents are experiential learners,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, pp. 19 632–19 642 on Pith
- [40] S. Zhong, Y . Lu, J. Ning, Y . Wan, L. Feng, Y . Ao, L. F. R. Ribeiro, M. Dreyer, S. Ammirati, and C. Xiong, “Skilllearnbench: Benchmarking continual learning methods for agent skill generation on real-world tasks,” arXiv preprint arXiv:2604.20087, 2026 on PithPith's reviewarXiv
- [41] Y . Zhou, Z. Zhang, Z. Cheng, S. Zhang, Q. Lan, Z. Chen, Z. Yang, Q. Xu, R. Chen, H. Wang, and S. Hu, “Skillgenbench: Benchmarking skill generation pipelines for LLM agents,”arXiv preprint arXiv:2605.18693, 2026 on PithPith's reviewarXiv
- [42] Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” inProceedings of the 11th International Conference on Learning Representations, 2023 on Pith
Cited by 3 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Guarding the Kingdom of Skill-Based Agents in the AI Dynasty of Computation: Control-Flow Integrity for Agent Skills cs.CR · Nov 2026
- Mining Agent Skills from Production Traces cs.AI · Oct 2026
- Component Replacement in LLM Agents: A Critical Scoping Review of Task Benefit and Attribution cs.AI · Sep 2026
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 2 sentences without a confident label
Uses the method or tool of 1 paper
-
Since each retained SkillsBench skill originally provides only one task query, we expand the query set for each selected skill through a two-stage augmentation process inspired by [ref] .
No Attack Required: Semantic Fuzzing for Specification Violations in Agent Skills · Evaluation Setup · 0.97
Uses data or values from 1 paper
-
We build our evaluation dataset based on SkillsBench [ref] , which contains 87 tasks associated with one user query, a skill document, a task execution environment and an outcome-based deterministic verifier.
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks · Evaluation Setup · 0.96
Cites as support 1 paper
-
Even when the LLM agent has demonstrated the capability to complete a task under the guidance of a skill before, it may fail to reproduce the correct execution consistently in later runs [ref] .
2601.06112 · Introduction · 0.98
Cites as background 6 papers
-
Some automatic prompt optimization and reflection-based approaches refine prompts, instructions, in-context memories, or skills based on execution feedback [ref] .
Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills · Introduction · 0.89
-
To guide multi-step planning and execution, skills have been introduced as reusable procedural knowledge for completing a class of tasks [ref] .
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks · Background · 0.99
-
To guide multi-step planning and execution, skills have been introduced as reusable procedural knowledge for completing a class of tasks [ref] .
SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks · Background · 0.99
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment