Representing Prompting Patterns with PDL: Compliance Agent Case Study

Anca Sailer; Hirokuni Kitahara; Louis Mandel; Mandana Vaziri; Martin Hirzel; Yuji Watanabe

REVIEW 3 major objections 4 minor 21 references

Representing Prompting Patterns with PDL: Compliance Agent Case Study

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A declarative prompt language called PDL lets developers restructure agent workflows, and a compliance-agent case study reports up to a 4x success gain over a canned implementation.

desk verdict A plausible engineering story with a thin quantitative core: the 4x improvement is suggestive but not established, and the comparison isn't clean enough to attribute to PDL. read the letter →

arxiv 2507.06396 v1 pith:3EO6WG3A submitted 2025-07-08 cs.AI cs.LGcs.PLcs.SE

Mandana Vaziri , Louis Mandel , Yuji Watanabe , Hirokuni Kitahara , Martin Hirzel , Anca Sailer This is my paper

classification cs.AIcs.LGcs.PLcs.SE

keywords PromptDeclarationLanguageengineeringLLMagentsReActpatternagenticprogrammingtoolcallreliabilitycomplianceautomationYAMLDSL

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Prompt engineering is hard because prompting patterns are buried inside code or fixed templates. The paper introduces the Prompt Declaration Language (PDL), a YAML-based representation that keeps prompts visible while letting developers compose model calls with ordinary code and tools. Using a real compliance agent, the paper claims that rewriting the standard prompting pattern in PDL, and customizing it for compact models, improved task success by up to 4x relative to a canned agent implementation. The gains come mainly from fewer tool-call failures, which matters because smaller, cheaper models are otherwise unreliable at agentic tasks.

What carries the argument

The central mechanism is PDL itself: a declarative YAML language in which a program is a sequence of blocks—text, message, model call, code, conditional—that implicitly accumulate a message context fed to LLM calls. That representation lets the developer express a prompting pattern as data, so the agent's loop can be restructured and outputs can be type-checked and constrained with JSON Schema. The structural change that carries the result is separating the natural-language thought from the structured tool-selection specification, plus a response parser that fixes format errors.

What would settle it

Run the original and PDL agents on the same evaluation suite with identical prompt text, model versions, tools, and repeated seeds, and also port the two-stage workflow into a framework that permits custom loops; if the 4x success-rate gap shrinks to within noise or disappears, the causal claim about the declarative representation is falsified.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that a declarative prompt representation is not just a readability aid but a performance lever: representing an agent's prompting pattern in PDL enables workflow-level customizations that canned agent frameworks do not allow, and those customizations materially improve task success. The case study splits the agent's single think-and-act step into a natural-language stage followed by a structured tool-selection stage, and adds a parser that repairs malformed tool calls. Across roughly 200 evaluation runs per condition, the paper reports consistent improvements for every model tested, with the compact model's success rate rising fourfold, driven by drops in cases where no tool was called.

Load-bearing premise

The comparison isolates the agent architecture, so the result depends on the assumption that the original and PDL versions shared identical models, tools, prompts, and evaluation runs, with no hidden difference in tuning effort or chance variation.

Editorial extensions

If this is right

Prompts and agent logic live in one readable YAML program, so developers can see and adjust the exact text and flow the model sees.
Agent workflows can be changed at the pattern level, not just by editing prompt strings, which is what lets a team split a ReAct-style loop into two stages.
Compact models become more viable for agentic tasks, since the two-stage design and parser reduce the syntax errors that previously blocked tool use.
Performance gains concentrate in tool-call reliability: PDL's improvements reduce cases where no tool is called, which the paper identifies as the main driver of higher task success.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same split-and-parse pattern may transfer to other domains where small models must emit structured actions, such as code generation or database queries, though the paper does not test this.
If the effect replicates, declarative pattern customization could become a cheaper alternative to scaling up model size for agentic workloads.
A direct test would be to implement the same custom workflow in a framework that permits workflow modification, using identical prompts and models, to check whether any custom workflow achieves the gain or whether PDL-specific features are required.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read for you on arXiv:2507.06396. The paper is a workshop-scale case study that claims up to 4x task-success improvement for a PDL-based compliance agent over a canned CrewAI ReAct baseline on ITBench. The mechanism—splitting the Think step into two stages and adding a tolerant parser for tool-call output—is sensible and matches known failure modes of small LLMs. That part is the real contribution: the case study shows what kind of customization helps, and the two-stage pattern plus parser are concrete, useful engineering ideas. The evaluation, though, is thin. No error bars, no raw counts, no statistical tests, and the described comparison in Section 3 says both versions use identical models and tools, but the PDL agent also gets a different prompt structure and a custom parser. The baseline's prompts and tuning effort are not described, so the measured gain cannot be cleanly attributed to PDL as a representation. The custom parser alone might explain a large chunk of the reduction in tool-call failures. The Sankey numbers are suggestive—22.4% no-tool-call down to 2.4% for gpt4o, 53.5% to 35.4% for granite—but the reported 4x success-rate improvement is not statistically established. There is also no release of code or data, which makes external checking hard. That said, I don't think the paper is unserious. The authors are clear that they tuned the PDL agent for compact models, and the failure analysis is grounded. The main weakness is methodological, not conceptual. A serious referee should ask for error bars, raw counts, full prompt and parser descriptions, and ideally an ablation where the custom parser is added to the CrewAI baseline. The paper would then be a useful data point for anyone building agent frameworks for small models. As it stands, it's a reasonable workshop paper but the headline number should not be cited as established. I'd send it to review if the venue has a low bar for preliminary empirical work, but it needs revision before I'd trust the 4x claim. My own verdict: reading-group maybe, wouldn't cite in next 12 months, but would accept for peer review.

Referee Report

3 major / 4 minor

Summary. The paper presents the Prompt Declaration Language (PDL), a YAML-based declarative representation for LLM prompts and agentic workflows, and reports a case study in which a PDL-based CISO compliance agent is compared with a CrewAI-based ReAct implementation. The authors claim up to 4x improvement in task success rate on ITBench, particularly for compact models, and attribute the gain to reduced tool-call failures enabled by PDL's customization of prompting patterns (e.g., splitting the Think step into Think1 and Think2 and adding a custom response parser). Sections 2 through 5 introduce PDL, describe the two architectures, present the evaluation in Figures 3 and 4, and discuss related work.

Significance. The paper addresses a real and timely problem: making agent prompting patterns customizable for smaller, cost-sensitive models. The PDL design itself builds on credible prior work (Vaziri et al., 2024; Spiess et al., 2025), and the case study uses a realistic compliance-agent workload on the public ITBench benchmark. The qualitative observation that a two-stage Think step reduces malformed JSON and tool-call failures is plausible and practically useful. However, the quantitative headline—up to 4x improvement—is not statistically supported as reported, and the comparison does not control for confounds such as prompt content, parser changes, and tuning effort. With stronger evidence, the result would be of clear practical value for compact LLM deployments.

major comments (3)

[Section 3, Figure 3] The headline claim of up to 4x improvement rests on a single bar chart with no raw success rates, no error bars, no repeated runs, and no statistical test. Since the absolute success rates are not reported, the 4x ratio may reflect a comparison of small percentages, and run-to-run variance over approximately 200 tests per condition could explain the difference. The authors should report exact counts per condition, confidence intervals, and ideally multiple seeds or a per-task breakdown.
[Section 3, 'Both versions use identical models and tools...'] The assertion that the two agents differ only in agent architecture is not substantiated. The PDL agent changes the prompting pattern (Think1/Think2), adds a custom Response Parser for malformed ActionSpec outputs, and may use different system and tool prompt text; the CrewAI baseline's prompts, parser, and tuning effort are not described. Consequently, the observed improvement cannot be attributed to PDL as a representation rather than to unequal engineering effort, to the custom parser alone, or to differences in evaluation subsets. A controlled comparison should use identical prompt text and tool descriptions, apply the same parser logic to both conditions, and document comparable tuning effort.
[Section 3, Figure 4] The mechanism claim that improvements stem primarily from reduced tool call failures is not quantitatively established. The Sankey diagrams report no-tool-call rates falling from 22.4% to 2.4% (gpt4o) and from 53.5% to 35.4% (granite3.2-8b), but they do not show raw counts, exact test numbers, or the joint distribution of tool-call accuracy and task success. The authors should provide a contingency table or equivalent analysis showing how much of the success-rate gain is explained by the reduced no-tool-call rate, and confirm that the task mix is identical across conditions.

minor comments (4)

[Section 4] The attribution 'DSPy (Zheng et al., 2023)' is incorrect; the DSPy reference should be Khattab et al. (2024), while Zheng et al. (2023) is the SGLang reference.
[Figures 3 and 4] Figure 3 has no axis labels or numeric values, and Figure 4's 'modelA' and 'modelB' labels are cryptic; please label axes and name the models explicitly in the captions.
[Section 3] The PDL source for the two agent variants is not provided, and no artifact link is given; including the full PDL and CrewAI configurations would substantially improve reproducibility and help verify the claimed parity of prompts and tools.
[Throughout] There are several typographical artifacts in the text, such as 'Y AML' and the code line 'W ik i p ed i a Ex c e pt i o n'; please proofread the final formatting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported 4x improvement is an empirical comparison between two agent implementations, not a result derived from PDL's own definitions.

full rationale

The paper's central claim is an empirical evaluation: a PDL-based CISO agent is compared against a CrewAI ReAct baseline on ITBench, and the measured success-rate improvement is reported. This is not a derivation from PDL semantics, and no equation or definition in the paper forces the outcome. Self-citations to Vaziri et al. (2024) and Spiess et al. (2025) describe PDL and prior use of PDL, but the case-study result is independently checkable through the benchmark and is not inferred from those citations. ITBench (Jha et al., 2025) is an external evaluation suite shared with some authors, but using it as a benchmark does not make the measured outcome equivalent to an input of the paper. The paper does not fit a parameter and call it a prediction, nor does it invoke a uniqueness theorem from the authors' prior work to rule out alternatives. Concerns about missing error bars, raw counts, or unequal tuning effort between the PDL agent and the canned baseline are threats to the validity of the causal attribution, but they are not circularity. Accordingly, no circular step is identified.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central result is an empirical comparison, not a derivation. There are no fitted numerical parameters. The main axioms are comparability of the two implementations and representativeness of the ITBench evaluation; these are domain assumptions stated in the text, not proven.

assumptions (3)

domain assumption The two agent implementations differ only in agent architecture, with identical models and tools, so performance differences are attributable to PDL.
Used implicitly in Section 3 to compare CrewAI ReAct vs PDL; if prompts or tool-handling differ, the 4x claim is confounded.
domain assumption ITBench tasks and approximately 200 tests per condition give a stable, representative estimate of compliance-agent performance.
The evaluation relies on ITBench without reporting confidence intervals or per-task results.
domain assumption JSON Schema type-checking and constrained decoding behave as described on the tested models.
Section 2 claims PDL uses constrained decoding; the paper does not verify that the models actually respect these constraints in the case study.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Representing Prompting Patterns with PDL: Compliance Agent Case Study." pith.science (2026). https://pith.science/paper/3EO6WG3A

@misc{pith2026250706396,
  author       = {Pith},
  title        = {Pith review of: Representing Prompting Patterns with PDL: Compliance Agent Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3EO6WG3A}},
  note         = {Machine review of arXiv:2507.06396}
}

read the original abstract

Prompt engineering for LLMs remains complex, with existing frameworks either hiding complexity behind restrictive APIs or providing inflexible canned patterns that resist customization -- making sophisticated agentic programming challenging. We present the Prompt Declaration Language (PDL), a novel approach to prompt representation that tackles this fundamental complexity by bringing prompts to the forefront, enabling manual and automatic prompt tuning while capturing the composition of LLM calls together with rule-based code and external tools. By abstracting away the plumbing for such compositions, PDL aims at improving programmer productivity while providing a declarative representation that is amenable to optimization. This paper demonstrates PDL's utility through a real-world case study of a compliance agent. Tuning the prompting pattern of this agent yielded up to 4x performance improvement compared to using a canned agent and prompt pattern.

Figures

Figures reproduced from arXiv: 2507.06396 by the authors.

**Figure 1.** Simple Tool Use in PDL Line 36 is a conditional. Although PDL is a YAML-based representation, it supports control structures such as conditionals and loops. It also supports modularity and reuse through function definitions and importing PDL code from other files or libraries. This design choice enables entire prompting patterns to be expressed in YAML, as opposed to being split apart into YAML and, for example, Py… view at source ↗

**Figure 2.** CISO Agent Architecture causes syntax errors in JSON output resulting in tool call failures or hallucination of tool names in the ActionSpec. As a solution to these problems, the CISO Agent developers devised a PDL-based agent architecture shown in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** CISO Agent Evaluation for Original (blue) and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Tool Call Success Rate Comparison (modelA: [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

21 extracted references · 15 canonical work pages

[1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...
[2]

LiteLLM , July 2025

BerryAI. LiteLLM , July 2025. URL https://docs.litellm.ai/

work page 2025
[3]

Prompting is programming: A query language for large language models

Beurer-Kellner, L., Fischer, M., and Vechev, M. Prompting is programming: A query language for large language models. In Conference on Programming Language Design and Implementation (PLDI), pp.\ 1946--1969, June 2023

work page 1946
[4]

Chase et al. , H. LangChain , July 2025. URL https://github.com/langchain-ai/langchain

work page 2025
[5]

Jha, S., Arora, R., Watanabe, Y., Yanagawa, T., Chen, Y., Clark, J., Bhavya, B., Verma, M., Kumar, H., Kitahara, H., Zheutlin, N., Takano, S., Pathak, D., George, F., Wu, X., Turkkan, B. O., Vanloo, G., Nidd, M., Dai, T., Chatterjee, O., Gupta, P., Samanta, S., Aggarwal, P., Lee, R., Murali, P., wook Ahn, J., Kar, D., Rahane, A., Fonseca, C., Paradkar, A....

work page 2025
[6]

V., Haq, S., Sharma, A., Joshi, T

Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., A, S. V., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. DSPy : Compiling declarative language model calls into self-improving pipelines. In International Conference on Learning Representations (ICLR), May 2024

work page 2024
[7]

Mernik, M., Heering, J., and Sloane, A. M. When and how to develop domain-specific languages. ACM Computing Surveys (CSUR), 37 0 (4): 0 316--344, 2005

work page 2005
[8]

Llama Stack , July 2025

Meta. Llama Stack , July 2025. URL https://github.com/meta-llama/llama-stack

work page 2025

Show all 21 references

[9]

\ guidance\ : A guidance language for controlling large language models, July 2025

Microsoft. \ guidance\ : A guidance language for controlling large language models, July 2025. URL https://github.com/guidance-ai/guidance

2025
[10]

CrewAI : Framework for orchestrating role-playing, autonomous AI agents, July 2025

Moura, J. CrewAI : Framework for orchestrating role-playing, autonomous AI agents, July 2025. URL https://github.com/crewAIInc/crewAI

2025
[11]

L., Suarez, F., Ugarte, M., and Vrgo c , D

Pezoa, F., Reutter, J. L., Suarez, F., Ugarte, M., and Vrgo c , D. Foundations of JSON schema. In International Conference on World Wide Web (WWW), pp.\ 263--273, April 2016

2016
[12]

and Zhang, B

Schluntz, E. and Zhang, B. Building effective agents, July 2025. URL https://www.anthropic.com/research/building-effective-agents

2025
[13]

AutoPDL : Automatic prompt optimization for LLM agents

Spiess, C., Vaziri, M., Mandel, L., and Hirzel, M. AutoPDL : Automatic prompt optimization for LLM agents. In Conference on Automated Machine Learning (AutoML), September 2025

2025
[14]

PDL : A declarative prompt programming language, October 2024

Vaziri, M., Mandel, L., Spiess, C., and Hirzel, M. PDL : A declarative prompt programming language, October 2024. URL http://arxiv.org/abs/2410.19135

2024 arXiv
[15]

Chain-of-thought prompting elicits reasoning in large language models

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems (NeurIPS), pp.\ 24824--24837, December 2022

2022
[16]

Willard, B. T. and Louf, R. Efficient guided generation for large language models, July 2023. URL https://arxiv.org/abs/2307.09702

2023 arXiv
[17]

H., White, R

Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., Liu, J., Awadallah, A. H., White, R. W., Burger, D., and Wang, C. AutoGen : Enabling next-gen LLM applications via multi-agent conversation, October 2023. URL https://arxiv.org/abs/2308.08155

2023 arXiv
[18]

Decoupling reasoning from observations for efficient augmented language models, September 2023

Xu, B., Peng, Z., Lei, B., Mukherjee, S., and Xu, D. Decoupling reasoning from observations for efficient augmented language models, September 2023. URL https://openreview.net/forum?id=CpgoO6j6W1

2023
[19]

R., and Cao, Y

Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K. R., and Cao, Y. ReAct : Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), May 2023

2023
[20]

EvoAgent : Towards automatic multi-agent generation via evolutionary algorithms, July 2024

Yuan, S., Song, K., Chen, J., Tan, X., Li, D., and Yang, D. EvoAgent : Towards automatic multi-agent generation via evolutionary algorithms, July 2024. URL https://arxiv.org/abs/2406.14228

2024 arXiv
[21]

H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J

Zheng, L., Yin, L., Xie, Z., Huang, J., Sun, C., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., Barrett, C., and Sheng, Y. Efficiently programming large language models using SGLang , December 2023. URL https://arxiv.org/abs/2312.07104

2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

[2] [2]

LiteLLM , July 2025

BerryAI. LiteLLM , July 2025. URL https://docs.litellm.ai/

work page 2025

[3] [3]

Prompting is programming: A query language for large language models

Beurer-Kellner, L., Fischer, M., and Vechev, M. Prompting is programming: A query language for large language models. In Conference on Programming Language Design and Implementation (PLDI), pp.\ 1946--1969, June 2023

work page 1946

[4] [4]

Chase et al. , H. LangChain , July 2025. URL https://github.com/langchain-ai/langchain

work page 2025

[5] [5]

Jha, S., Arora, R., Watanabe, Y., Yanagawa, T., Chen, Y., Clark, J., Bhavya, B., Verma, M., Kumar, H., Kitahara, H., Zheutlin, N., Takano, S., Pathak, D., George, F., Wu, X., Turkkan, B. O., Vanloo, G., Nidd, M., Dai, T., Chatterjee, O., Gupta, P., Samanta, S., Aggarwal, P., Lee, R., Murali, P., wook Ahn, J., Kar, D., Rahane, A., Fonseca, C., Paradkar, A....

work page 2025

[6] [6]

V., Haq, S., Sharma, A., Joshi, T

Khattab, O., Singhvi, A., Maheshwari, P., Zhang, Z., Santhanam, K., A, S. V., Haq, S., Sharma, A., Joshi, T. T., Moazam, H., Miller, H., Zaharia, M., and Potts, C. DSPy : Compiling declarative language model calls into self-improving pipelines. In International Conference on Learning Representations (ICLR), May 2024

work page 2024

[7] [7]

Mernik, M., Heering, J., and Sloane, A. M. When and how to develop domain-specific languages. ACM Computing Surveys (CSUR), 37 0 (4): 0 316--344, 2005

work page 2005

[8] [8]

Llama Stack , July 2025

Meta. Llama Stack , July 2025. URL https://github.com/meta-llama/llama-stack

work page 2025