Pith.

Artificial Intelligence · arXiv:2608.09253 v1 · submitted 10 Aug 2026 · 42 references

SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance

You Lu, Xinyu Huang, Bihuan Chen, Xin Peng

The paper says

“LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably.”

more of the abstract

Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 11 Aug 2026 · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A solid runtime-assurance system with a credible 24.1% success-rate gain, held back mainly by unvalidated LLM extraction and missing artifacts. This line is wrong

3 major objections · 6 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the 42 references say.

The paper says, at III-B, III-C, III-D

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The runtime assurance mechanism treats the LLM-extracted logical actions and failure patterns as ground truth, but the validation described in Sections III-B and III-C is only structural (unique step IDs, acyclic dependencies, valid regular expressions, well-formed fields). The FSM in Section III-D marks step si complete only when all extracted logical actions are matched, and the procedure checker denies an …

more of objection 1

… action when it matches a failure pattern; if the GPT-5.4 parser or miner hallucinates a logical action, a step can be marked complete before the required operation occurs, and if a necessary logical action is omitted, a correct execution can be denied termination or spuriously re-planned. The paper reports no assessment of whether extracted logical actions are necessary or sufficient for each step, whether failure patterns correspond to actual failure causes, or whether suggestions and warnings contradict the skill document. Because the reported 24.1% improvement is produced through these interventions, the headline result is conditional on the semantic quality of the extracted guidance, which is never measured.

The paper says, at IV-B, Table II

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

The evaluation in Table II measures only end-to-end task success rates and their standard deviations; it does not report any analysis of the correctness of the checker's individual decisions. Without data on the number of interventions, the rate of false-positive failure-pattern triggers, whether denials were followed by successful re-planning, and whether termination denials occurred on executions that would have succeeded without intervention, the causal contribution of the proposed runtime assurance mechanism is not established.

more of objection 2

The single case study in Figure 6 is illustrative but does not substitute for a systematic analysis of the checker's precision and recall.

The paper says, at IV-A

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The executability filter in Section IV-A retains only the 15 skills that each of the four agent-model configurations can complete at least once in five executions, and the paper does not report how many skills were excluded or analyze the sensitivity of the headline improvement to this choice.

more of objection 3

This selection biases the evaluation toward skills on which agents already show partial competence, so the 24.1% average improvement should be stated as conditional on this subpopulation rather than as a general property of skill execution. The paper should either report results on the full set with appropriate controls or explicitly bound the claims to the filtered set and discuss the direction of the potential bias.

6 minor objections

The paper says, at III-C, IV-A

Pith says: minor objection 1 of 6

The origin of the historical successful and failed traces used to seed the initial runtime guidance is not specified; the paper should state whether these are the five filtering runs from Section IV-A or a separate collection.

The paper says, at IV-D, Figure 8

Pith says: minor objection 2 of 6

The self-evolving analysis in Figure 8 is shown for a single skill and configuration; a curve averaged over all 15 skills, with confidence bands, should be included in the main text to support the general claim of self-evolution.

show 4 more

The paper says, at Table III

Pith says: minor objection 3 of 6

The 'Token Cost' column in Table III should state the unit explicitly; the values appear to be token counts, but this is not stated in the caption or the table header.

The paper says, at V

Pith says: minor objection 4 of 6

The paired t-test statement in Section V lacks important details: the pairing unit, the number of pairs, the test statistic, and whether any multiple-comparison correction was applied. Please provide this information or release the per-run results needed to verify it.

The paper says, at III-B, III-C, IV-A

Pith says: minor objection 5 of 6

Since GPT-5.4 is used both as the parser/miner and as one of the backbone models (Codex with GPT-5.4), the paper should discuss any potential shared-model confound, even if the evaluation queries are held out, to reassure readers that the extracted guidance is not implicitly tuned to the same model's stylistic preferences.

The paper says, at VI

Pith says: minor objection 6 of 6

The claim of being 'the first' runtime assurance framework for skill-oriented agent execution is a strong novelty statement; consider softening it or providing a more systematic search of the literature to support the claim.

What would settle it

Release the 15 runtime-guidance instances and have two independent annotators label every logical action, failure pattern, suggestion, and warning as correct, incorrect, or incomplete against the skill document and a sample of evolution traces. Then rebuild SkillSentry with the corrected guidance, removing or fixing all incorrect or incomplete items, and rerun the held-out Qtest evaluations. Pre-specify a threshold: if the corrected average success rate is more than 5 percentage points below the reported 77.7%, semantic extraction quality is load-bearing; if it matches, the assurance mechanism itself, rather than the specific mined content, drives the improvement and the concern does not land.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 11 Aug 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 1 min 20 s · 17,981 tokens read, 7,383 written
  3. Editor wrote the verdict above 22 s · 18,735 tokens read, 2,361 written
  4. Referee major revision: 3 major objections, 6 minor 1 min 6 s · 18,156 tokens read, 6,431 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Read SkillSentry. What is new here is the combination: a DSL that holds both the parsed skill specification and mined execution experience, wrapped around the agent loop with step-aware checking. That is a genuinely useful design, and the evaluation is more careful than most: 15 skills, four agent-model configurations, held-out task instances, five repeated runs, ablations, and a self-evolution curve. The headline claim — average success from 62.6% to 77.7%, with standard deviation down 41.1% — looks internally consistent. The per-skill table shows improvement in every one of 60 configurations, and the paired t-test is reported. The overhead numbers are believable: 7.8% more turns, 0.8% of execution time in its own computation.

Credit where due: the ablation isolates on-enter guidance, failure patterns, and step-level delivery, and the system-prompt comparison is a fair control. The Fig. 6 case study is illustrative but not load-bearing.

Soft spots, in order of size. First, semantic extraction is load-bearing and only structurally validated. The GPT-5.4 parser and miner decide what counts as a logical action, a failure pattern, and a useful suggestion; validation checks unique step IDs, acyclic dependencies, and valid regexes. A hallucinated logical action can mark a step complete before the required operation happens, or a missing one can deny termination on a correct execution. Both change the interventions behind the 24.1% gain. No parser/miner output or guidance instance is released, so this cannot be checked. Second, the skill filter keeps only 15 skills where all four agent-model pairs succeeded at least once in five runs. That is a sensible way to study instability rather than capability, but it softens the general benchmark claim. Third, there is no direct comparison with AgentSpec, ShieldAgent, or similar runtime enforcement. The paper justifies the omission, but the reader cannot tell how much of the gain is skill-specific versus generic guardrailing. Fourth, minor: GPT-5.4 generates paraphrases, mines traces, and also serves as a backbone; the held-out split keeps this from explaining the results, but it is a slight self-reference.

Who this is for: people building runtime layers for LLM agents and researchers measuring skill reliability. It deserves a serious referee. I would ask for artifact release, semantic spot-checks of extracted guidance, and at least one direct baseline before treating 24.1% as a settled benchmark. The architecture and main effect are credible.

The strongest objection

The most load-bearing link in the argument is the unvalidated semantic correctness of the LLM-generated runtime guidance. In Sec. III-B, a GPT-5.4 parser converts each natural-language skill document into DSL steps, dependencies, constraints, and termination requirements; in Sec. III-C, a GPT-5.4 miner converts successful/failed traces into logical actions, suggestions, warnings, and failure patterns. The runtime checker in Sec. III-D then builds an FSM in which a step is completed only when all extracted logical actions match, and it denies or replans on failure patterns. The paper's validation is structural only: unique step IDs, dependency acyclicity, valid regular expressions, and well-formed fields. Nothing checks whether a parsed logical action is actually necessary or sufficient for the step, whether a failure pattern truly reflects a failure cause, or whether a suggestion contradicts the skill document. A hallucinated logical action can mark a step complete before the required operation occurs; a missing logical action can deny termination on a correct execution and trigger spurious replanning. Both failure modes change the interventions that produce the reported 24.1% gain. Since no parser/miner output or artifact is released, the headline result cannot be separated from extraction quality.

The reader identified the same concern, and the stress test confirms it is the correct soft spot. The architecture is coherent and the empirical evaluation is internally consistent: recomputing Table II row averages reproduces the aggregate 62.6% to 77.7% transition, a relative improvement of 24.1%. The per-skill positive results are extensive, and the ablation supports the claimed mechanism. The missing piece is not arithmetic; it is an unverified semantic layer on which the runtime FSM depends. Structural validation cannot catch a false-positive or false-negative logical action, and both would change the interventions that produce the measured gains. A human-annotation plus corrected-guidance rerun would distinguish 'runtime assurance works because the extracted guidance is semantically right' from 'runtime assurance works even with imperfect extraction.' Because this is an addressable empirical gap rather than a demonstrated error, the reader's CONDITIONAL verdict remains appropriate; artifact release and a semantic audit should be conditions. I would not move to REJECT.

The referee's report

If the claimed effects hold, SkillSentry would demonstrate that runtime assurance, rather than model retraining or prompt rewriting, can substantially improve the reliability of skill execution in LLM agents. The paper has notable strengths: a broad evaluation covering 60 skill-model combinations, held-out test queries with task-level splits, five repeated runs per condition, ablations isolating components, a transfer study, and a fail-open design that reduces risk of disruption. The use of deterministic verifiers for trace labeling is also a positive. However, the central claim is conditional on the semantic correctness of LLM-generated runtime guidance, which is not validated beyond structural checks; the paper also lacks intervention-level analysis that would show the checker's decisions themselves are reliable. These gaps make the current evidence suggestive rather than conclusive, though the outcome is important enough to warrant further work.

Pith's reading of the paper (a machine's summary, not the authors')

SkillSentry claims that many residual failures of LLM agents are not capability failures but execution-instability failures, and that a runtime assurance wrapper can catch and correct them without retraining or rewriting prompts. It builds structured runtime guidance for each skill by parsing the skill document and mining historical successful and failed traces, then wraps the agent loop with a procedure checker and a termination checker that nudge the agent back on procedure. Across 15 skills, two agent runtimes, and four backbone models, the paper reports average task success rising from 62.6% to 77.7%, a relative gain of 24.1%, while the standard deviation across repeated runs falls by 41.1% and the checker itself uses about 0.8% of total execution time. If that holds, runtime assurance becomes a practical complement to model training and prompt engineering for making LLM agents dependable.

What carries the argument. The load-bearing object is the runtime guidance DSL: each skill becomes a list of steps, each with a unique identifier, dependency list, tool and parameter constraints, logical actions with regular-expression action patterns, on-enter suggestions and warnings, and failure patterns with reasons, plus a termination list marking steps that must be completed. An LLM-based parser extracts the specification from the skill document, an LLM-based miner fills the experience fields from labelled traces, and at runtime the guidance is compiled into a finite-state machine whose states are skill steps and whose completion condition is that all logical actions of a step have been matched. The procedure checker advances this FSM, delivers step guidance, and temporarily denies actions that violate dependencies or match failure patterns, while the termination checker blocks final output until the required steps are done.

Core claim. The paper's central claim is that unreliable skill execution is largely procedural: agents skip required steps, execute steps out of order, or use wrong parameters, even when they have demonstrated the capability to finish the task before. SkillSentry encodes what a skill is supposed to do in a small domain-specific language, seeds that encoding from the skill document as a specification and from successful and failed traces as execution experience, and then enforces it live. It matches each planned action against expected logical actions, temporarily denies actions that skip dependencies or repeat historically failing patterns, injects step-level suggestions and warnings at the right moment, and refuses termination until all required steps are complete. The evaluation claims this raises average success from 62.6% to 77.7% and cuts repeated-run standard deviation by 41.1%, with gains on every tested configuration and the largest relative gains on skills with the weakest baselines.

The premise it rests on. The framework's entire benefit rests on the language model that reads skill documents and execution traces turning them into accurate runtime guidance; the paper checks only formatting-level properties, not whether the extracted steps, constraints, and failure patterns actually mean what the skill requires.

If the paper is right (Pith's inferences, not the authors')

If the gains hold, an agent that already knows a skill can be made reliable by wrapping rather than retraining: the same capability, once verified, becomes reproducible across similar queries and repeated runs.

Because the checker is fail-open and adds only about 0.8% of execution time, it can sit in production agent loops without becoming the bottleneck.

Step-level guidance delivery beats putting all guidance in the system prompt by 5.8%, implying that presenting guidance in the right execution context matters as much as the guidance content.

Runtime guidance evolved with one backbone model retains roughly 94.2% of its native benefit when transferred to another backbone under the same agent, so assurance artifacts are partially portable.

Self-evolution over ten iterations delivers most of its benefit early, suggesting the guidance converges quickly and a modest trace collection budget suffices.

The paper evaluates only skills on which each agent succeeded at least once, so the method is aimed at instability rather than capability; extending it to never-succeeded skills would test whether its experience mining can bootstrap without positive traces.

Because the parser and miner are themselves LLM calls, the framework's ceiling likely tracks the extraction model's fidelity; a cheaper model might reproduce the results only if the extracted guidance stays semantically correct.

SkillSentry is complementary to offline prompt optimization: it corrects transient deviations at runtime, while prompt rewriting changes the distribution those deviations arise from, so combining both could push reliability higher than either alone.

The termination checker only knows the required steps the parser extracted; adding a check on the final output's semantic correctness would catch cases where all steps ran but the result is still wrong.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.11 · human participants 0.04 · controlled comparison 0.35 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.33 · mentions code or data availability 0.03

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. background 1.00 0.05 0.02 asserted 0.99
Although skills provide reusable procedural knowledge, agents may still execute them unreliably. background 0.77 0.07 0.02 hedged possibility 0.99
Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. background 0.94 0.10 0.03 hedged possibility 0.97
Such instability limits the practical reliability of LLM agents. background 0.91 0.06 0.03 asserted 1.00
To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. contribution 0.96 0.86 0.07 asserted 0.99
SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. method 0.99 0.87 0.03 asserted 1.00
It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. method 1.00 0.78 0.03 asserted 0.99
We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. method 0.95 0.95 0.03 asserted 0.86
Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs. result 1.00 0.97 0.90 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,975 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (9)
Figure 2 of the paper
Figure 2. Example Cases of Skill Execution Failure
Figure 3 of the paper
Figure 3. Approach Overview of SKILLSENTRY To illustrate this problem, we show two cases of Claude Code [4] under the guidance of the skill for macroeconomic timeseries detrending. As illustrated by Case-1 in
Figure 5 of the paper
Figure 5. An Example of Runtime Guidance B. Skill Specification Extraction Given the DSL, SKILLSENTRY first extracts the skill specifi￾cation from the corresponding skill document. A skill document is written in natural language. Therefore, SKILLSENTRY uses an LLM-based parser with GPT-5.4 [23] to transform each skill document into the specification fields defined by the DSL. Specifically, given a skill document Dskill, the parser extracts four types of specification information. First, it identifies the expected procedure steps, assigning each step a unique identifier and a short natural-lang
Figure 6 of the paper
Figure 6. An Example of Runtime Assurance by SKILLSENTRY macroeconomic-timeseries-detrending skill. Upon entering each skill step, SKILLSENTRY provides the corresponding sugges￾tions and warnings from the runtime guidance. After loading and inspecting the data, the agent attempts to proceed directly to Convert to Real Terms without completing the required preceding step, i.e., Harmonize Frequency. SKILLSENTRY detects the deviation from the skill procedure and temporarily denies the action. It returns a targeted hint identifying the missing step. Guided by this intervention, the agent re-plans
Figure 8 of the paper
Figure 8. illustrates the self-evolving process of SKILLSENTRY for macroeconomic-timeseries-detrending under Claude Code with Claude-Haiku-4.5. The shaded area denotes the standard deviation (SD) across five repeated evolution runs. As more traces are incorporated, the average task success rate improves by about 58.2% after 10 iterations. The largest gains occur in the early stages, as newly collected traces enrich action patterns, suggestions, and warnings. The improvement becomes relatively small during later stages, indicating that the runtime guidance tends to stabilize on the current…
Figure 9 of the paper
Figure 9. Results of the Generalization Evaluation

Showing 6 of the paper's 9 figures. Figures are the authors', reproduced from the paper.

References (42)
  1. [1] Anonymous GitHub. (2026) SkillSentry. Accessed on May 11, 2026. [Online]. Available: https://anonymous.4open.science/r/skillsentry-865C on Pith
  2. [2] Anthropic, “Introducing claude haiku 4.5,” https://www.anthropic.com/ news/claude-haiku-4-5, Oct. 2025, accessed on June 8, 2026 on Pith
  3. [3] ——. (2025) Writing effective tools for agents — with agents. Accessed on May 13, 2026. [Online]. Available: https://www.anthropic.com/engi neering/writing-tools-for-agents on Pith
  4. [4] ——, “Claude code,” https://code.claude.com/docs/en/overview, 2026, accessed on June 8, 2026 on Pith
  5. [5] ——, “Claude code docs: Hooks reference,” https://code.claude.com/do cs/en/hooks, 2026, accessed on June 13, 2026 on Pith
  6. [6] ——, “Introducing claude opus 4.6,” https://www.anthropic.com/news/c laude-opus-4-6, Feb. 2026, accessed on June 8, 2026 on Pith
  7. [7] Y . Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnonet al., “Constitutional ai: Harmlessness from ai feedback,”arXiv preprint arXiv:2212.08073, 2022 on PithPith's reviewarXiv
  8. [8] I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” inProceedings of the IEEE/ACM 47th International Conference on Software Engineering, 2025, pp. 2188– 2200 on Pith
  9. [9] I. Bouzenia and M. Pradel, “Understanding software engineering agents: A study of thought-action-result trajectories,” inProceedings of the 2025 40th IEEE/ACM International Conference on Automated Software Engineering, 2025, pp. 2846–2857 on Pith
  10. [10] J. F. Box, “Guinness, gosset, fisher, and small samples,”Statistical Science, pp. 45–52, 1987 on Pith
  11. [11] Z. Chen, M. Kang, and B. Li, “Shieldagent: Shielding agents via verifiable safety policy reasoning,” inProceedings of the International Conference on Machine Learning, 2025, pp. 8313–8344 on Pith
  12. [12] Google. (2023) Google Gemini. Accessed on May 13, 2026. [Online]. Available: https://gemini.google.com/ on Pith
  13. [13] A. Gupta, “Reliabilitybench: Evaluating llm agent reliability under production-like stress conditions,”arXiv preprint arXiv:2601.06112, 2026 on PitharXiv
  14. [14] X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,”ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, pp. 1–79, 2024 on Pith
  15. [15] T. Kumar, A. Tripathy, G. Saranathan, M. Foltin, S. Bhattacharya, S. Hinchley, D. M. Bahls, D. Brookshire, L. Kaplan, and R. W. Wisniewski, “Infrastructuresentinel: Policy enforced guardrails for secure mcp-driven infrastructure agents,” inProceedings of the AAAI Conference on... on Pith
  16. [16] X. Li, W. Chen, Y . Liu, S. Zheng, X. Chen, Y . He, Y . Li, B. You, H. Shen, J. Sunet al., “Skillsbench: Benchmarking how well agent skills work across diverse tasks,”arXiv preprint arXiv:2602.12670, 2026 on PithPith's reviewarXiv
  17. [17] Y . Li, H. Wen, Y . Chen, H. Liu, Y . Tian, and Y . Feng, “No attack required: Semantic fuzzing for specification violations in agent skills,” arXiv preprint arXiv:2605.13044, 2026 on PithPith's reviewarXiv
  18. [18] J. Ni, Y . Liu, X. Liu, Y . Sun, M. Zhou, P. Cheng, D. Wang, E. Zhao, X. Jiang, and G. Jiang, “Trace2skill: Distill trajectory-local lessons into transferable agent skills,”arXiv preprint arXiv:2603.25158, 2026 on PithPith's reviewarXiv
  19. [19] OpenAI. (2023) Introducing ChatGPT. Accessed on May 13, 2026. [Online]. Available: https://openai.com/blog/chatgpt on Pith
  20. [20] OpenAI, “Introducing gpt-5.2,” https://openai.com/index/introducing-gpt -5-2/, Dec. 2025, accessed on June 8, 2026 on Pith
show 22 more
  1. [21] ——, “Codex,” https://developers.openai.com/codex/, 2026, accessed on June 8, 2026 on Pith
  2. [22] ——, “Codex docs: Hooks,” https://developers.openai.com/codex/hooks, 2026, accessed on June 13, 2026 on Pith
  3. [23] ——, “Introducing gpt-5.4,” https://openai.com/index/introducing-gpt-5 -4/, Mar. 2026, accessed on June 8, 2026 on Pith
  4. [24] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in neural information processing systems, vol. 35, pp. 27 730–27 744, 2022 on Pith
  5. [25] OWASP Foundation, “Improper error handling,” https://owasp.org/ww w-community/Improper Error Handling, 2026, accessed on June 25, 2026 on Pith
  6. [26] R. Pryzant, D. Iter, J. Li, Y . T. Lee, C. Zhu, and M. Zeng, “Automatic prompt optimization with “gradient descent” and beam search,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 7957–7968 on Pith
  7. [27] Y . Qin, S. Liang, Y . Ye, K. Zhu, L. Yan, Y . Lu, Y . Lin, X. Cong, X. Tang, B. Qianet al., “Toolllm: Facilitating large language models to master 16000+ real-world apis,” inProceedings of the International Conference on Learning Representations, vol. 2024, 2024, pp. 9695–9717 on Pith
  8. [28] T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, E. Hambro, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,”Advances in Neural Informa- tion Processing Systems, vol. 36, pp. 68 539–68 551, 2023 on Pith
  9. [29] N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: language agents with verbal reinforcement learning,” in Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023, pp. 8634–8652 on Pith
  10. [30] G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large language models,”arXiv preprint arXiv:2305.16291, 2023 on PithPith's reviewarXiv
  11. [31] H. Wang, C. M. Poskitt, and J. Sun, “Agentspec: Customizable runtime enforcement for safe and reliable LLM agents,” inProceedings of the 48th IEEE/ACM International Conference on Software Engineering, 2026 on Pith
  12. [32] Q. Wu, G. Bansal, J. Zhang, Y . Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liuet al., “Autogen: Enabling next-gen llm applications via multi-agent conversations,” inProceedings of the 1st Conference on Language Modeling, 2024 on Pith
  13. [33] Z. Xue, Y . Zhao, S. Wang, K. Chen, and H. Wang, “A characterization study of bugs in llm agent workflow orchestration frameworks,” in Proceedings of the 2025 40th IEEE/ACM International Conference on Automated Software Engineering, 2025, pp. 3369–3380 on Pith
  14. [34] C. Yang, X. Wang, Y . Lu, H. Liu, Q. V . Le, D. Zhou, and X. Chen, “Large language models as optimizers,” inProceedings of the 12th International Conference on Learning Representations, 2024 on Pith
  15. [35] J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024 on Pith
  16. [36] S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022 on PithPith's reviewarXiv
  17. [37] A. Zeng, X. Lv, Z. Hou, Z. Du, Q. Zheng, B. Chen, D. Yin, C. Ge, C. Huang, C. Xieet al., “Glm-5: from vibe coding to agentic engineering,” arXiv preprint arXiv:2602.15763, 2026 on PithPith's reviewarXiv
  18. [38] Y . Zhang, H. Ruan, Z. Fan, and A. Roychoudhury, “Autocoderover: Autonomous program improvement,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2024, pp. 1592–1604 on Pith
  19. [39] A. Zhao, D. Huang, Q. Xu, M. Lin, Y .-J. Liu, and G. Huang, “Expel: Llm agents are experiential learners,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 17, 2024, pp. 19 632–19 642 on Pith
  20. [40] S. Zhong, Y . Lu, J. Ning, Y . Wan, L. Feng, Y . Ao, L. F. R. Ribeiro, M. Dreyer, S. Ammirati, and C. Xiong, “Skilllearnbench: Benchmarking continual learning methods for agent skill generation on real-world tasks,” arXiv preprint arXiv:2604.20087, 2026 on PithPith's reviewarXiv
  21. [41] Y . Zhou, Z. Zhang, Z. Cheng, S. Zhang, Q. Lan, Z. Chen, Z. Yang, Q. Xu, R. Chen, H. Wang, and S. Hu, “Skillgenbench: Benchmarking skill generation pipelines for LLM agents,”arXiv preprint arXiv:2605.18693, 2026 on PithPith's reviewarXiv
  22. [42] Y . Zhou, A. I. Muresanu, Z. Han, K. Paster, S. Pitis, H. Chan, and J. Ba, “Large language models are human-level prompt engineers,” inProceedings of the 11th International Conference on Learning Representations, 2023 on Pith
Cited by 3 Pith papers
How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 2 sentences without a confident label

Uses the method or tool of 1 paper
Uses data or values from 1 paper
Cites as support 1 paper
  • Even when the LLM agent has demonstrated the capability to complete a task under the guidance of a skill before, it may fail to reproduce the correct execution consistently in later runs [ref] .

    2601.06112 · Introduction · 0.98

Cites as background 6 papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment