Pith. sign in

REVIEW 6 major objections 5 minor 68 references

Prompt programming should separate what a task is from how the model runs it, and the same Imprompt program can be compiled in two ways that measurably change accuracy and cost.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 07:01 UTC pith:UVXSRUC3

load-bearing objection A genuinely formal prompt-language framework with useful static-analysis and IR artifacts, but the empirical claims run ahead of the evidence and the central 'compiler defines semantics' thesis is model-relative in ways the paper concedes. the 6 major comments →

arxiv 2607.22683 v1 pith:UVXSRUC3 submitted 2026-07-13 cs.AI cs.CLcs.PL

Imprompt: A Language Framework for Prompt Programming

Chentian Wu , Shengyuan Yang , Adithya Murali This is my paper
classification cs.AI cs.CLcs.PL
keywords prompt programmingprompt compilationlanguage modelsconstrained decodingtype systemscompilation semanticsImprompt
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that prompt programming has been mixing two distinct concerns: describing a task and choosing the low-level details of how a language model executes it. To separate them, it introduces Imprompt, a deliberately small language whose programs contain only task logic, plus a type system that turns output expectations into string constraints. The same Imprompt program can then be given a meaning by a compiler, and the paper formalizes two compilers: a baseline that emits one big prompt together with a static, type-derived output constraint, and a stepwise compiler that emits a sequence of model interactions through an explicit stack. The empirical claim is that the choice of compiler changes accuracy and cost in measurable ways, with the stepwise compiler usually ahead but not always. If the claim holds, prompt engineering becomes a compile-time concern: users write the task once and future systems choose the execution strategy.

Core claim

The central claim is that a prompt program's semantics is defined by its compilation, not by the language alone. The paper develops this by defining Imprompt with a minimal set of constructs—reason, cast-as-type, typed get-from-template, conditionals, loops, and say—and then giving the language two compilers. The baseline compiler flattens a program into a single prompt with XML-like structure and uses a static analysis of the program's types to generate one global string constraint; a soundness theorem states that any string the model produces from the compiled prompt satisfies that constraint. The stepwise compiler lowers the program to a stack-based intermediate representation whose opera

What carries the argument

The load-bearing object is the pair (Imprompt program, compiler). Imprompt is a small imperative-style language with a type system in which every type is a string constraint: Dyn is no constraint, Bool is exactly true/false, and a user-defined type wraps a natural-language description with a decidable formal constraint. The two compilers then define what a program means: the baseline compiler produces a single prompt plus a global constraint derived by a static analysis that walks the program and joins constraints at control-flow merges, and the stepwise compiler produces a stack-based instruction sequence (post, gen, cgen, branch, loop, print, etc.) whose operational semantics fixes the ord

Load-bearing premise

The load-bearing premise is that a real language model responds to a compiled prompt the way the formal model assumes—one stable decoding step per prompt under the given constraint—rather than according to unpredictable model-specific formatting preferences; the paper itself concedes this premise is only partially met, since prompt formats that help one model family can hurt another.

What would settle it

Run a fixed Imprompt program with one if/else branch through the baseline and stepwise compilers on the same model, with the same temperature, decoding seed, and inputs, and measure accuracy and token cost per mode over several hundred cases. If the two modes are statistically indistinguishable on both metrics, the paper's central empirical claim—that compilation strategy changes LM behavior—fails. A sharper variant: repeat after tuning the prompt representation per model; if tuning erases the gap, then the compiler's measured effect is a proxy for model-specific format preference, which would

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • The same source prompt program can be compiled in multiple ways without editing the task logic, so users no longer hand-code which prompting technique to apply at each call.
  • Compiler choice becomes a tunable compile-time parameter, analogous to an optimization level, and future systems can search over strategies automatically.
  • Type annotations become enforceable output guarantees: the baseline compiler's static analysis derives a constraint that every produced string satisfies, so output formats can be checked before runtime rather than hoped for.
  • Externalizing state in the stepwise compiler relieves the model from remembering variable bindings across steps; this is why the paper finds stepwise usually more accurate and cheaper on structured tasks.
  • No single strategy dominates—baseline can win on some tasks and models—so the compilation space should be studied as a design space rather than reduced to a single best practice.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if compilation strategies really define semantics, then compiler choice should be adapted to input data as well as to model family—a short input may decode better stepwise while a long input may need a single-pass baseline.
  • Editorial inference: the paper's admission that prompt formats must be model-aware suggests the decoupling thesis holds most cleanly once models converge on prompt conventions; until then, a compiler may need to select a prompt representation per model family.
  • Editorial inference: a testable extension is to treat compiler selection as a hyperparameter and run a search over strategies for a fixed program across models; if accuracy variance from compiler choice is comparable to variance from prompt wording, compilation choice deserves the same optimization attention as prompt engineering.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper introduces Imprompt, a small declarative prompt-programming language intended to decouple task descriptions from low-level execution choices such as prompting formats, decoding constraints, and hyperparameters. It formalizes a core language with a type system, defines two compilation strategies — a baseline one-shot strategy with a global type-directed constraint, and a stepwise strategy compiling to an intermediate representation with an operational semantics — and implements these in a Python DSL, Imprompt-Py. The evaluation compares baseline, baseline-without-constraints, stepwise, and natural-language prompting on two case studies: SARA statutory reasoning and PromptPex data-labeling/transformation tasks, across three instruction-tuned models. The paper's central claim is that prompt programs should contain only task descriptions and that compilation strategies define the language's semantics; a secondary empirical claim is that stepwise compilation generally improves accuracy and cost over baseline and natural prompting.

Significance. If the claims are properly qualified, this is a useful contribution to prompt-programming foundations: it is one of the few works to give prompt compilation an explicit syntax, type-directed constraint generation, and an operational IR, rather than treating prompting as an ad hoc artifact. The formal core is mostly coherent and the two compilation strategies are genuinely instructive. The paper also honestly identifies limitations in its concluding section. However, several formal gaps and empirical overstatements need to be addressed before the central claims are supported: the main soundness theorem is close to tautological, the loop rule is under-specified, the formal core does not cover the functions used in the evaluation, and the empirical evidence for 'stepwise generally wins' is not sustained by the paper's own tables.

major comments (6)
  1. [§4.2, Theorem 4.3] Theorem 4.3 is, as stated, a definitional property of constrained decoding rather than a soundness theorem about the compiler or the type system. In Definition 4.1, an execution of BComp(P, σ) is by definition decode(D[JP Kσ], C[P]); any correct constrained decoder returns a string satisfying C[P]. The proof-sketch appeal to induction on P is therefore vacuous. This matters because the paper presents Theorem 4.3 as the formal output-format guarantee of the type-directed analysis. Please rephrase it as a compliance property of the constrained-decoding procedure, or provide a substantive theorem relating C[P] to a pre-existing semantics of Imprompt programs.
  2. [§4.2, Figure 16, rule C-While] Rule C-While uses typeof(x_i) to widen the constraint environment, but the constraint-generation judgment Δ⊢c⇝φ⊣Δ' does not carry a typing context Γ. The environment Δ maps variables to string constraints, and a constraint does not determine a type: for instance, the singleton constraint {'true'} could arise from a Bool-typed variable or from a Dyn-typed expression via E-Str. Thus the widening maps ⌊Δ⌋ and ⌈Δ⌉ are not well-defined as written. This directly affects Theorem 4.2's well-definedness for programs containing while loops, including the text-to-p example in Figure 30. Please either thread Γ through the constraint-generation rules or define and prove a type-recovery operation on Δ.
  3. [§3.2 / §5.1 / §6.3] The formal core language and IR contain no function definitions or calls, yet the implementation and evaluation rely on first-class @dsl functions, cross-section function calls (e.g., Figure 27b), and 'standard handling for function arguments and function calls' mentioned in §6.2. Formal claims about compilation semantics therefore do not apply to the actual benchmark programs being evaluated. If functions are an extension of the core, they should be formally specified; otherwise the evaluation should be presented as covering an unformalized superset of Imprompt, and the formal contributions should be scoped accordingly.
  4. [§6.5, RQ1; Table 2; Figure 32] The claim that 'stepwise generally achieves higher accuracy and lower cost' is not supported by the reported data. Rankings flip by task and model: in Table 2, Stepwise reaches 94.4% on text-to-p with Phi3.5-Mini but only 27.8% with Gemma2-9B; on elements with Llama3.2-1B, Baseline achieves 55.6% while Stepwise achieves 0.0%; and in aggregate on Gemma2-9B, Natural mode (74.6%) beats Stepwise (70.1%). With three runs and no variance or significance testing, 'generally' carries no statistical meaning. Please report per-model, per-task confidence intervals or other uncertainty measures, and state the conclusion more narrowly (e.g., 'stepwise tends to help on constrained-output tasks for some models').
  5. [§3.4 / §5.2 / §6.5 RQ4] The formal semantics in Section 5.2 are parameterized by a model and decoding policy via D[π] and decode(D[π], φ). The paper's own RQ4 acknowledges that compilation 'should ideally be model-aware' and that prompt formats differ across model families. Consequently, the statement that 'every compiler defines a semantics for Imprompt' holds only relative to a fixed model/decoder pair; the same compiler induces different semantics for different models, as Table 2 illustrates. This does not invalidate the decoupling principle, but the semantics should be explicitly presented as model-indexed, and the conclusion in Section 9 should not claim a single language-level semantics without this qualification.
  6. [§6.5, RQ2] The Baseline-versus-Baseline-WC comparison does not isolate the value of Imprompt's type-directed static constraint analysis. Removing all output constraints is expected to hurt on grammar-bound tasks such as speech-tag and text-to-p; the comparison mostly demonstrates that constrained decoding helps. To support the claim that the type system's analysis itself adds value, compare against a hand-written output grammar, or against a variant that applies constraints only at final say statements while leaving the static analysis disabled, or perform an ablation that keeps prompt structure fixed and varies only the precision of C[P].
minor comments (5)
  1. [§3.3, Lemma 3.1] The lemma title contains a typo: 'Uniqeness' should be 'Uniqueness'.
  2. [§2] The line 'Detok(:)𝑅→Σ∗' appears malformed; the intended type is Detok : R → Σ∗.
  3. [§6.2] The evaluation reports means over three runs without standard deviations or per-trial results. Please include dispersion measures, especially for the Table 2 numbers where accuracy differences are large and cell sizes are small (n = 6, 12, 13, 18).
  4. [§6.1] The paper states that code 'will be made available' in the project repository, which is not sufficient for reproducibility review. Please include a usable anonymized repository link or supplement the submission with the implementation.
  5. [§6.5, RQ3] The claim that 'there exists at least one Imprompt compilation strategy that outperforms natural-language prompting' is weak and per-model/per-task; it should be stated with the specific strategy and model for which it holds, or removed in favor of a stronger quantified statement.

Circularity Check

1 steps flagged

Empirical comparisons are external and non-forced; the only circularity is a formal soundness theorem that restates constrained decoding by definition.

specific steps
  1. self definitional [Section 4, Definition 4.1 and Theorem 4.3]
    "We define an execution of the compiled program as a constrained decoding process of D[JP Kσ] under the constraint C[P]. ... Theorem 4.3 (Constraint Soundness). If an execution of the compiled program BComp(P,σ) produces a string s, then s |= C[P]."

    By Definition 4.1, BComp(P,σ) = (D[JP Kσ], C[P]) and execution is defined as constrained decoding under C[P]. Section 2 defines constrained decoding as a process that returns a response ω with Detok(ω) |= φ. Hence any output of an execution of BComp(P,σ) satisfies C[P] by definition; the soundness conclusion does not depend on the program rewriting, Lemma 3.1, or the induction. It restates the construction, so it cannot serve as independent evidence that the type-directed constraint analysis is sound.

full rationale

The central empirical claims are not circular: RQ1 compares two genuinely different compilation strategies on external benchmarks (SARA, PromptPex), and the paper explicitly acknowledges that the ranking is not uniform and that compilation should ideally be model-aware. RQ2 is a controlled comparison between constrained and unconstrained versions of the same compiled prompt, which is an empirical treatment/control experiment rather than a prediction forced by construction. There are no load-bearing self-citations; all citations used to justify design choices are external. The only notable definitional step is Theorem 4.3, whose conclusion is already guaranteed by the definition of execution as constrained decoding under C[P]. This is a minor self-definitional restatement, not a driver of the paper's substantive empirical or language-design conclusions. Overall, the paper is largely self-contained against external benchmarks, so the appropriate verdict is low-severity circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 2 invented entities

No numeric parameter is fitted to the empirical results; the only hand-chosen numbers are standard decoding settings that affect all modes equally (temperature, top-p, timeout). The heavier baggage is conceptual: the idealized decode function abstracts away exactly the stochasticity and model-specific format sensitivity that the paper's own results show matter, and the backward-simulation stance makes 'semantics' non-falsifiable until an equivalence theory exists. The constraint domain Δ and Imprompt-IR are paper-internal constructs whose utility is established only by in-paper evaluation.

free parameters (3)
  • temperature = 0.7
    Fixed decoding temperature for all runs; hand-chosen standard value, not fitted to target results, but it shapes every reported distribution and ranking.
  • top-p = 0.9
    Fixed nucleus-sampling threshold for all runs; hand-chosen standard value.
  • per-call timeout = 120 s
    Any LM call exceeding 120s counts as failure in the success metric; hand-chosen and directly affects reported success/coverage numbers.
axioms (5)
  • standard math String constraints used in compilation are decidable and closed under union, concatenation, and Kleene star.
    Invoked in the proof sketch of Theorem 4.2 (closure property of decidable languages) and used to build C[P] via C-If (disjunction), C-Seq (concatenation), and C-While (Kleene star) in Figure 16.
  • domain assumption An LM is modeled as a distribution D[s] over completions, and decode(D[π], φ) returns a string satisfying φ.
    Section 2 and rules I-Gen/I-Cgen (Figure 22); the entire compilation semantics abstracts LM behavior to this idealized function, ignoring stochasticity, instruction-following reliability, and format sensitivity.
  • ad hoc to paper Every compiler defines an acceptable semantics for an Imprompt program (backward simulation).
    Section 3.4 states 'every compiler defines a semantics for Imprompt' and defers a formal equivalence theory to future work; this interpretive stance is adopted by the authors, not derived.
  • domain assumption The baseline prompt rendering using XML-like tags is a suitable universal surface for LMs.
    Section 4.1 cites Anthropic best practices [3]; Section 6.5 RQ4 concedes that the best prompting format differs for Gemma ([15]), so the assumption is empirically fragile and the authors acknowledge model-specific compilation would be needed.
  • domain assumption Type casts and Bool-valued reason responses yield exactly the strings in Mod(C[τ]) when they succeed.
    Stepwise branch rules I-Branch-T/I-Branch-F require the stack top to be exactly 'true' or 'false'; Section 7 acknowledges casts 'returning a null value when no reasonable cast exists,' meaning LM failure is folded into the value domain rather than modeled.
invented entities (2)
  • Imprompt language (reason/get/say, if/while, Dyn/Bool/Ty types) no independent evidence
    purpose: The core artifact: lets users write task descriptions decoupled from execution details.
    The language's value is demonstrated only through the paper's own case studies; there is no third-party adoption or external replication yet, and the paper treats its design as a first step.
  • Imprompt-IR (post, gen, cgen, push, lookup, pop, dup, print, cat, template, and/or/not, br, while) no independent evidence
    purpose: Minimal compilation target for the stepwise strategy, making LM interactions explicit.
    Paper-internal construct; its adequacy as 'a set of low-level instructions' is asserted and illustrated but not benchmarked against other IR proposals, and the authors themselves leave context algebras to future work.

pith-pipeline@v1.3.0-alltime-deepseek · 27680 in / 17787 out tokens · 156221 ms · 2026-08-02T07:01:36.874788+00:00 · methodology

0 comments
read the original abstract

With the unprecedented success of Language Models (LMs), the science of Prompt Engineering has evolved the powerful idea of Prompt Programming, where prompts are treated as a programmable control surface for describing complex tasks and leveraging LM capabilities. However, existing prompt programming frameworks suffer from various complexities and inelegances, which make them hard to utilize in practice for effectively describing tasks. We propose Imprompt, a new language framework for the study and practice of prompt programming. We undertake a foundational investigation of prompt programming, and contend that prompt programs must contain only the task descriptions and must be decoupled from lower-level 'execution' details. We further develop this position by illustrating structured prompting as a combination of prompt programming and prompt program 'compilation'. We exemplify this view by formally defining two compilers for Imprompt programs. We then explore the idea of typing for prompt programs and draw a correspondence between type checking and constrained decoding. Finally, we implement our compilers and type checkers and evaluate them on a variety of case studies. We believe our work contributes programming-language foundations toward the emerging area of prompt programming.

Figures

Figures reproduced from arXiv: 2607.22683 by Adithya Murali, Chentian Wu, Shengyuan Yang.

Figure 1
Figure 1. Figure 1: RAG application from DSPy homepage [13] therefore cannot assign formal semantics to prompt languages in a direct, meaningful way. Instead, the semantics of a prompt language is defined by its compilation. Different compilation strategies induce potentially different semantics, and these semantic differences can directly affect LM behavior. To the best of our knowledge, no prior work on prompt programming h… view at source ↗
Figure 2
Figure 2. Figure 2: Motivating example: a Part-of-Speech (PoS) tagging prompt. [ [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: A Imprompt program for Fair-Use Copyright Analysis. Constructs with LM interactions. The main distinction between prompt programming languages and classical programming languages is that the former includes primitives for interacting with LMs. Imprompt has three such constructs. (1) reason 𝑒 takes an expression 𝑒 as a step of task description and returns the LM response. For example, in line 1, the overall… view at source ↗
Figure 4
Figure 4. Figure 4: Global environment for Figure 2b in JSON Variables and environments. Imprompt uses let and get to declare internal variables, and also supports external variables from a global environment immutable to Imprompt programs. External variables are not declared in the program but may be referenced in tem￾plates (e.g., INPUT on line 2). This design re￾flects the fact that practical prompts are often not plain te… view at source ↗
Figure 5
Figure 5. Figure 5: presents the core syntax of Imprompt. Like Imp, we intentionally keep Imprompt small: it uses a minimal set of constructs sufficient to express structured prompting procedures. We focus here on the formal presentation of the syntax, and readers seeking intuition may refer to Section 3.1 for an introduction to the language constructs. Lit 𝑙 ::= 𝑠 ∈ Σ ∗ | true | false Tmpl 𝑡 ::= 𝑠 ∈ Σ ∗ | {x} | 𝑡1𝑡2 Type 𝜏 :… view at source ↗
Figure 6
Figure 6. Figure 6: Typing rules for Imprompt expressions T-Skip Γ ⊢ skip ⊣ Γ T-Let Γ ⊢ 𝑒 : 𝜏 𝑥 ∉ dom(Γ) Γ ⊢ let 𝑥 = 𝑒 ⊣ Γ, 𝑥 : 𝜏 T-Let-2 Γ ⊢ 𝑒 : 𝜏 Γ(𝑥) ≡ 𝜏 Γ ⊢ let 𝑥 = 𝑒 ⊣ Γ T-Get Γ ⊢ 𝑒 : 𝜏 ′ 𝑥 ∉ dom(Γ) Γ ⊢ get 𝑥: 𝜏 from 𝑡 ⊣ Γ, 𝑥 : 𝜏 T-Say Γ ⊢ 𝑒 : 𝜏 Γ ⊢ say 𝑒 ⊣ Γ T-Seq Γ ⊢ 𝑐1 ⊣ Γ1 Γ1 ⊢ 𝑐2 ⊣ Γ2 Γ ⊢ 𝑐1 𝑐2 ⊣ Γ2 T-If Γ ⊢ 𝑒 : Bool Γ ⊢ 𝑐1 ⊣ Γ1 Γ ⊢ 𝑐2 ⊣ Γ2 Γ ⊢ if 𝑒 { 𝑐1 } else { 𝑐2 } ⊣ Γ T-While Γ ⊢ 𝑒 : Bool Γ ⊢ 𝑐 ⊣ Γ1 Γ ⊢ while 𝑒 … view at source ↗
Figure 7
Figure 7. Figure 7: Well-formation rules for statements in Imprompt The type information obtained from Imprompt’s type checking serves two purposes. ❶ The primary purpose is to assist compilation, which we elaborate in detail in Section 4. ❷ The secondary purpose is to help programmers describe their intentions more precisely, to increase the likelihood of correct program execution under the stochastic nature of LMs [PITH_FU… view at source ↗
Figure 8
Figure 8. Figure 8: Types for precise intent description. For example, [PITH_FULL_IMAGE:figures/full_fig_p008_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Semantics via backward simulation. In Section 4 and Section 5, we present two compilation strategies for Imprompt. Each induces a distinct semantics for the language. In Section 6, we define evaluation metrics such as success rate and test coverage and compare the performance of the two strategies across different case studies. The compilation space of Imprompt, however, is far from exhausted, and we discu… view at source ↗
Figure 10
Figure 10. Figure 10: Baseline compilation workflow JDynK = a value JBoolK = a boolean value JTy<𝜔, 𝜑>K = 𝜔 [PITH_FULL_IMAGE:figures/full_fig_p010_10.png] view at source ↗
Figure 13
Figure 13. Figure 13: J·K : Stmt → Σ ∗ Note that expressions and statements need not be closed: free variables are compiled to named placeholders in the prompt artifact. XML-like tags are used to help the LM distinguish different parts of the prompt artifact. Such structured prompts tend to perform better on specific models according to contemporary records on best practices [3] ❶ Transforming types. The first step is defining… view at source ↗
Figure 14
Figure 14. Figure 14: Γ is too coarse for constraints At a high level, the type constraint is computed by statically modeling the execution of programs and iteratively replacing all occurrences of variables to their tightest constraints. However, using only typeof(·) would be too coarse for an informative constraint, because our type system is designed to be minimal. For example in [PITH_FULL_IMAGE:figures/full_fig_p011_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Generating constraints from expressions These rules are intuitive. E-Str and E-Tmpl specify that string parts of templates are constrained precisely. E-Var and E-Var-Ext indicate that we first try to retrieve a variable’s constraint from Δ, and if it is not found there, we retrieve it from the global environment 𝜎. E-BOp and E-Not specify that the constraint for boolean expressions can be precise at most … view at source ↗
Figure 16
Figure 16. Figure 16: shows the rules for generating constraints from statements. C-Skip Δ ⊢ skip ⇝ ⊥ ⊣ Δ C-Let Δ ⊢ 𝑒 ⇝ 𝜑 Δ ⊢ let 𝑥 = 𝑒 ⇝ ⊥ ⊣ Δ[𝑥 ↦→ 𝜑] C-Get Δ ⊢ get 𝑥: 𝜏 from 𝑡 ⇝ ⊥ ⊣ Δ[𝑥 ↦→ C [𝜏 ] ] C-Say Δ ⊢ 𝑒 ⇝ 𝜑 Δ ⊢ say 𝑒 ⇝ 𝜑 ⊣ Δ C-Seq Δ ⊢ 𝑐1 ⇝ 𝜑1 ⊣ Δ1 Δ1 ⊢ 𝑐2 ⇝ 𝜑2 ⊣ Δ2 Δ ⊢ 𝑐1 𝑐2 ⇝ 𝜑1 · 𝜑2 ⊣ Δ2 C-If Δ ⊢ 𝑒 ⇝ C [Bool] Δ ⊢ 𝑐1 ⇝ 𝜑1 ⊣ Δ1 Δ ⊢ 𝑐2 ⇝ 𝜑2 ⊣ Δ2 Δ ⊢ if 𝑒 { 𝑐1 } else { 𝑐2 } ⇝ 𝜑1 ∨ 𝜑2 ⊣ Δ1 ⊔ Δ2 C-While ⌊Δ⌋ ≜ Δ[𝑥𝑖 ↦→ C [t… view at source ↗
Figure 17
Figure 17. Figure 17: Definition of modified variables With [PITH_FULL_IMAGE:figures/full_fig_p013_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: illustrates the overall workflow of stepwise compilation and execution. Stepwise Compiler transformation IMPROMPT program 𝑃 § 5.1 𝑃 𝜎 decode and store Execution output Input data 𝜎 push “Hello” ; post; cgen 𝜑; IMPROMPT-IR update distribution Stack true false ... 𝑆 LM [PITH_FULL_IMAGE:figures/full_fig_p013_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Intermediate representation for stepwise compilation. [PITH_FULL_IMAGE:figures/full_fig_p013_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: J·K : Expr → IR for expressions. Jlet 𝑥 = 𝑒K = J𝑒K; pop 𝑥 Jget 𝑥: 𝜏 from 𝑡K = template 𝑡; post; cgen 𝜏; pop 𝑥 Jsay 𝑒K = J𝑒K; print Jif 𝑒 { 𝑐1 } else { 𝑐2 }K = J𝑒K; br J𝑐1K J𝑐2K Jwhile 𝑒 { 𝑐 }K = while J𝑒K J𝑐K J𝑐1 𝑐2K = J𝑐1K; J𝑐2K [PITH_FULL_IMAGE:figures/full_fig_p014_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: J·K : Stmt → IR for statements. The rules in [PITH_FULL_IMAGE:figures/full_fig_p014_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Operational semantics for executing the stepwise IR. [PITH_FULL_IMAGE:figures/full_fig_p015_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: Evaluation order matters [PITH_FULL_IMAGE:figures/full_fig_p015_23.png] view at source ↗
Figure 24
Figure 24. Figure 24: Implementation workflow of Imprompt-Py. Types and constraints. As mentioned in Section 3, user-defined types Ty<𝜔, 𝜑> are implemented as a Python class extending a base class Type, with the textual description 𝜔 given by the class docstring and the constraint 𝜑 given by a method constraint. Grammar constraints returned by the method constraint are described in Lark [47] and enforced during decoding by llg… view at source ↗
Figure 26
Figure 26. Figure 26: Constrained decoding [PITH_FULL_IMAGE:figures/full_fig_p017_26.png] view at source ↗
Figure 27
Figure 27. Figure 27: Dependencies in SARA benchmark SARA case facts + question Statute runner Natural Baseline Baseline-WC Stepwise Adjudication layer function result → benchmark answer Final label true / false / unanswered [PITH_FULL_IMAGE:figures/full_fig_p019_27.png] view at source ↗
Figure 28
Figure 28. Figure 28: SARA evaluation workflow. issue one additional request to the same model, asking it to convert the function result into the final boolean benchmark answer (or unanswered). In SARA, we do not use a single external expert model for this adjudication step. because our goal is to evaluate end-to-end task performance in a model-specific way, and introducing a separate judge model would confound that measuremen… view at source ↗
Figure 29
Figure 29. Figure 29: SARA results. while Baseline-WC degrades sharply, indicating that type-directed constraints are doing substantial work. Stepwise is also cheaper, using about 343–373 median tokens versus roughly 1.2k–1.5k for Baseline, with median latency of 162–329ms versus 307ms–2.38s. 6.4 Case Study 2: Data Labeling and Transformation Our examples in this section are drawn from PromptPex [46], an automated tool for gen… view at source ↗
Figure 30
Figure 30. Figure 30: Example: text-to-p core function Compared to Section 6.3, tasks in this case study are shorter and their outputs are much more tightly constrained. In SARA, the main challenge is long-range statutory reasoning over larger prompts. In PromptPex, the challenge is often to satisfy strong local string constraints. For example, the short program in [PITH_FULL_IMAGE:figures/full_fig_p020_30.png] view at source ↗
Figure 31
Figure 31. Figure 31: Type PTag in text-to-p problem use GPT-5.4 as a judge model: given the test input/output and the task description, it determines whether the output meets the specification. Since these tasks are short, success and coverage are both 100%, so we report only accuracy. For each test, we run three trials and report the mean accuracy. Natural Baseline Baseline-WC Stepwise Llama3.2-1B-Instruct Phi3.5-Mini-Instru… view at source ↗
Figure 32
Figure 32. Figure 32: PromptPex overall results. Task Llama3.2-1B-Instruct Phi3.5-Mini-Instruct Gemma2-9B-IT N B B-WC S N B B-WC S N B B-WC S classify-input-text (𝑛 = 6) 66.7 33.3 0.0 33.3 83.3 66.7 66.7 100.0 100.0 100.0 33.3 100.0 elements (𝑛 = 18) 33.3 55.6 0.0 0.0 38.9 55.6 55.6 72.2 44.4 72.2 66.7 77.8 extract-names (𝑛 = 12) 0.0 8.3 0.0 8.3 33.3 50.0 16.7 75.0 83.3 58.3 33.3 83.3 speech-tag (𝑛 = 13) 0.0 0.0 0.0 7.7 38.5 2… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 1 canonical work pages

  1. [1]

    DotTXT AI. 2025. Outlines. Website. https://dottxt-ai.github.io/outlines/latest/ DSL for controlling large language models, accessed on 05-01-2025

    2025

  2. [2]

    Guidance AI. 2025. Guidance: A guidance language for controlling large language models. GitHub repository. https://github.com/guidance-ai/guidance Accessed on 02-05-2025

    2025

  3. [3]

    Anthropic. 2024. Prompt Engineering Best Practices for Claude: Using XML Tags. https://platform.claude.com/docs/en/ build-with-claude/prompt-engineering/claude-prompting-best-practices#structure-prompts-with-xml-tags Online; accessed 17 March 2026

    2024

  4. [4]

    Debangshu Banerjee, Tarun Suresh, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. 2025. CRANE: Reasoning with constrained LLM generation. arXiv:2502.09061 [cs.PL] https://arxiv.org/abs/2502.09061

    Pith/arXiv arXiv 2025

  5. [5]

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023. Prompting Is Programming: A Query Language for Large Language Models.Proc. ACM Program. Lang.7, PLDI, Article 186 (June 2023), 24 pages. doi:10.1145/3591300

    doi:10.1145/3591300 2023

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. InProceedings of the 34th International Conference on Neural Information Processing Systems. 1877–1901

    2020

  7. [7]

    Alexandra Butoi, Robin Chan, Ryan Cotterell, William Merrill, Franz Nowak, Clemente Pasti, Lena Strobl, and Anej Svete. 2024. Computational Expressivity of Neural Language Models. InProceedings of the 62nd Annual Meeting of the Imprompt: A Language Framework for Prompt Programming 25 Association for Computational Linguistics (Volume 5: Tutorial Abstracts)...

    2024

  8. [8]

    Harrison Chase. 2022. LangChain. GitHub repository. https://github.com/langchain-ai/langchain

    2022

  9. [9]

    Daniel Deutsch, Shyam Upadhyay, and Dan Roth. 2019. A General-Purpose Algorithm for Constrained Sequential Inference. InProceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), Mohit Bansal and Aline Villavicencio (Eds.). Association for Computational Linguistics, Hong Kong, China, 482–492. doi:10.18653/v1/K19- 1045

    work page doi:10.18653/v1/k19- 2019

  10. [10]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Jill Burstein, Christy ...

    doi:10.18653/v1/n19-1423 2019

  11. [11]

    Honghua Dong, Qidong Su, Yubo Gao, Zhaoyu Li, Yangjun Ruan, Gennady Pekhimenko, Chris J Maddison, and Xujie Si. 2024. APPL: A Prompt Programming Language for Harmonious Integration of Programs and Large Language Model Prompts.arXiv preprint arXiv:2406.13161(2024)

    Pith/arXiv arXiv 2024

  12. [12]

    Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, and Tianqi Chen

    Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ruihang Lai, Ziyi Xu, Yilong Zhao, and Tianqi Chen. 2025. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv:2411.15100 [cs.CL] https: //arxiv.org/abs/2411.15100

    Pith/arXiv arXiv 2025

  13. [13]

    DSPy. 2026. Building AI Applications by Customizing DSPy Modules. https://dspy.ai/tutorials/custom_module/. DSPy homepage

    2026

  14. [14]

    Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making Pre-trained Language Models Better Few-shot Learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (Ed...

    doi:10.18653/v1/2021.acl-long.295 2021

  15. [15]

    Google. 2025. FunctionGemma: Formatting and Best Practices. https://ai.google.dev/gemma/docs/functiongemma/ formatting-and-best-practices. Last updated Dec 18, 2025. Accessed 2026-03-18

    2025

  16. [16]

    Chris Hokamp and Qun Liu. 2017. Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Regina Barzilay and Min-Yen Kan (Eds.). Association for Computational Linguistics, Vancouver, Canada, 1535–1546. doi:10.18653/v1/P17-1141

    doi:10.18653/v1/p17-1141 2017

  17. [17]

    Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2020. A Dataset for Statutory Reasoning in Tax Law Entailment and Question Answering. InProceedings of the Natural Legal Language Processing Workshop 2020 co- located with the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD 2020), Virtual Workshop, August 24...

    2020

  18. [18]

    Madhav Kanda, Shubham Ugare, and Sasa Misailovic. 2026. RefineStat: Efficient Exploration for Probabilistic Program Synthesis. arXiv:2509.01082 [cs.LG] https://arxiv.org/abs/2509.01082

    Pith/arXiv arXiv 2026

  19. [19]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines.The Twelfth International Conference on Learning Representations

    2024

  20. [20]

    Terry Koo, Frederick Liu, and Luheng He. 2024. Automata-based constraints for language model decoding. InFirst Conference on Language Modeling. https://openreview.net/forum?id=BDBdblmyzY

    2024

  21. [21]

    Michael Kuchnik, Virginia Smith, and George Amvrosiadis. 2023. Validating Large Language Models with ReLM. arXiv:2211.15458 [cs.LG] https://arxiv.org/abs/2211.15458

    Pith/arXiv arXiv 2023

  22. [22]

    LangChain. 2025. LangGraph. Documentation. https://langchain-ai.github.io/langgraph/

    2025

  23. [23]

    Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K

    Alexander K. Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K. Mansinghka. 2023. Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs. arXiv:2306.03081 [cs.AI] https://arxiv.org/abs/2306.03081

    Pith/arXiv arXiv 2023

  24. [24]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. InProceedings of the 34th International Conference on Neural Information Processing Systems(Van...

    2020

  25. [25]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2023. Holistic evaluation of language models.Transactions on Machine Learning Research(2023). https://openreview.net/forum?id=iO4LZIBl7R

    2023

  26. [26]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.ACM Comput. Surv.55, 9, 26 Chentian Wu, Shengyuan Yang, and Adithya Murali Article 195 (Jan. 2023), 35 pages. doi:10.1145/3560815

    doi:10.1145/3560815 2023

  27. [27]

    Lew, Tim Vieira, and Timothy J

    João Loula, Benjamin LeBrun, Li Du, Ben Lipkin, Clemente Pasti, Gabriel Grand, Tianyu Liu, Yahya Emara, Marjorie Freedman, Jason Eisner, Ryan Cotterell, Vikash Mansinghka, Alexander K. Lew, Tim Vieira, and Timothy J. O’Donnell

  28. [28]

    Ximing Lu, Peter West, Rowan Zellers, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2021. NeuroLogic Decoding: (Un)supervised Neural Text Generation with Predicate Logic Constraints. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Kristina Toutanova, An...

    2021

  29. [29]

    2025.From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps

    Yuetian Mao, Junjie He, and Chunyang Chen. 2025.From Prompts to Templates: A Systematic Prompt Template Analysis for Real-world LLMapps. Association for Computing Machinery, New York, NY, USA, 75–86. https://doi.org/10.1145/ 3696630.3728533

    arXiv 2025

  30. [30]

    Stephen Mell, Konstantinos Kallas, Steve Zdancewic, and Osbert Bastani. 2025. Opportunistically Parallel Lambda Calculus.Proc. ACM Program. Lang.9, OOPSLA2, Article 365 (Oct. 2025), 27 pages. doi:10.1145/3763143

    doi:10.1145/3763143 2025

  31. [31]

    Hayes, Xin Qiu, Babak Hodjat, and Risto Miikkulainen

    Elliot Meyerson, Giuseppe Paolo, Roberto Dailey, Hormoz Shahrzad, Olivier Francon, Conor F. Hayes, Xin Qiu, Babak Hodjat, and Risto Miikkulainen. 2025. Solving a Million-Step LLM Task with Zero Errors. arXiv:2511.09030 [cs.AI] https://arxiv.org/abs/2511.09030

    arXiv 2025

  32. [32]

    Elliot Meyerson and Xin Qiu. 2025. Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives. (2025). arXiv:2502.04358 [cs.CL] https://arxiv.org/abs/2502.04358

    Pith/arXiv arXiv 2025

  33. [33]

    Microsoft. 2025. GenAIScript: Generative AI Scripting. Website. https://microsoft.github.io/genaiscript/ Accessed on 01-09-2025

    2025

  34. [34]

    Microsoft. 2025. Prompt-flow: industry. GitHub repository. https://github.com/microsoft/promptflow Accessed on 01-05-2025

    2025

  35. [35]

    Microsoft. 2025. PromptPex: Test Generation for Prompts. https://github.com/microsoft/promptpex. GitHub repository, accessed 2026-03-18

    2025

  36. [36]

    Kanghee Park, Timothy Zhou, and Loris D’Antoni. 2025. Flexible and Efficient Grammar-Constrained Decoding. In Forty-second International Conference on Machine Learning. https://openreview.net/forum?id=L6CYAzpO1k

    2025

  37. [37]

    Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. InAdvances in Neural Information Processing Systems, Vol. 34. 11054–11070

    2021

  38. [38]

    Gabriel Poesia, Alex Polozov, Vu Le, Ashish Tiwari, Gustavo Soares, Christopher Meek, and Sumit Gulwani. 2022. Synchromesh: Reliable Code Generation from Pre-trained Language Models. InInternational Conference on Learning Representations. https://openreview.net/forum?id=KmtVD97J43e

    2022

  39. [39]

    Matt Post and David Vilar. 2018. Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), Marilyn Walker, Heng Ji, and Amanda Stent (Eds.). Associa...

    doi:10.18653/v1/n18-1119 2018

  40. [40]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. https://api.semanticscholar.org/CorpusID:160025533

    2019

  41. [41]

    Kiran Ramnath, Kang Zhou, Sheng Guan, Soumya Smruti Mishra, Xuan Qi, Zhengyuan Shen, Shuai Wang, Sangmin Woo, Sullam Jeoung, Yawei Wang, Haozhu Wang, Han Ding, Yuzhe Lu, Zhichao Xu, Yun Zhou, Balasubramaniam Srinivasan, Qiaojing Yan, Yueyan Chen, Haibo Ding, Panpan Xu, and Lin Lee Cheong. 2025. A Systematic Survey of Automatic Prompt Optimization Techniqu...

    2025

  42. [42]

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. 2025. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications. arXiv:2402.07927 [cs.AI] https://arxiv.org/abs/2402.07927

    Pith/arXiv arXiv 2025

  43. [43]

    Tobias Schnabel and Jennifer Neville. 2024. Symbolic Prompt Program Search: A Structure-Aware Approach to Efficient Compile-Time Prompt Optimization. InFindings of the Association for Computational Linguistics: EMNLP 2024. 670–686. https://aclanthology.org/2024.findings-emnlp.37

    2024

  44. [44]

    Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chenglei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander Da Costa, Saloni Gupta, Megan L. Rogers, Inna Goncearenco, G...

    Pith/arXiv arXiv 2024

  45. [45]

    SgfdDttt. 2024. Statutory Reasoning Assessment (SARA). https://github.com/SgfdDttt/sara. GitHub repository, accessed 2026-03-18

    2024

  46. [46]

    Reshabh K Sharma, Jonathan De Halleux, Shraddha Barke, and Benjamin Zorn. 2025. PromptPex: Automatic Test Generation for Language Model Prompts. arXiv:2503.05070 [cs.SE] https://arxiv.org/abs/2503.05070

    arXiv 2025

  47. [47]

    2024.Lark Parser

    Erez Shinan. 2024.Lark Parser. https://github.com/lark-parser/lark

    2024

  48. [48]

    Tarun Suresh, Debangshu Banerjee, Shubham Ugare, Sasa Misailovic, and Gagandeep Singh. 2025. DINGO: Constrained Inference for Diffusion LLMs. arXiv:2505.23061 [cs.LG] https://arxiv.org/abs/2505.23061

    Pith/arXiv arXiv 2025

  49. [49]

    Zhi Rui Tam, Cheng-Kuang Wu, Yi-Lin Tsai, Chieh-Yen Lin, Hung-yi Lee, and Yun-Nung Chen. 2024. Let Me Speak Freely? A Study On The Impact Of Format Restrictions On Large Language Model Performance.. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Franck Dernoncourt, Daniel Preoţiuc-Pietro, and Anas...

    doi:10.18653/v1/2024.emnlp-industry.91 2024

  50. [50]

    Shubham Ugare, Rohan Gumaste, Tarun Suresh, Gagandeep Singh, and Sasa Misailovic. 2025. IterGen: Iterative Semantic-aware Structured LLM Generation with Backtracking. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=ac93gRzxxV

    2025

  51. [51]

    Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. 2025. SynCode: LLM Generation with Grammar Augmentation.Transactions on Machine Learning Research(2025). https://openreview.net/forum?id= HiUZtgAPoH

    2025

  52. [52]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  53. [53]

    Mandana Vaziri, Louis Mandel, Claudio Spiess, and Martin Hirzel. 2024. PDL: A Declarative Prompt Programming Language. arXiv:2410.19135 [cs.AI] https://arxiv.org/abs/2410.19135

    Pith/arXiv arXiv 2024

  54. [54]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171 [cs.CL] https://arxiv.org/abs/2203.11171

    Pith/arXiv arXiv 2023

  55. [55]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum?id=1PL1NIMMrw

    2023

  56. [56]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, Vol. 35. 24824–24837

    2022

  57. [57]

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2023. Large Language Models are Better Reasoners with Self-Verification. arXiv:2212.09561 [cs.AI] https://arxiv.org/abs/2212.09561

    Pith/arXiv arXiv 2023

  58. [58]

    Brandon T Willard and Rémi Louf. 2023. Efficient Guided Generation for LLMs.arXiv preprint arXiv:2307.09702(2023)

    Pith/arXiv arXiv 2023

  59. [59]

    Glynn Winskel. 1993. The formal semantics of programming languages.International Journal of Foundations of Computer Science(1993). https://api.semanticscholar.org/CorpusID:117797097

    1993

  60. [60]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-...

    2020

  61. [61]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: deliberate problem solving with large language models. InProceedings of the 37th International Conference on Neural Information Processing Systems(New Orleans, LA, USA)(NIPS ’23). Curran Associates Inc., Red Hook, NY, USA, Artic...

    2023

  62. [62]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629 [cs.CL] https://arxiv.org/abs/2210.03629

    Pith/arXiv arXiv 2023

  63. [63]

    Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023. Automatic Chain of Thought Prompting in Large Language Models. InThe Eleventh International Conference on Learning Representations. https://openreview.net/forum? id=5NTt8GFjUHkr

    2023

  64. [64]

    Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. InInternational Conference on Machine Learning. PMLR, 12697–12706

    2021

  65. [65]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs.arXiv preprint arXiv:2312.07104(2024). https://arxiv.org/abs/2312.07104 28 Chentian Wu, Shengyuan Yang, and A...

    Pith/arXiv arXiv 2024

  66. [66]

    Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Neil Zhenqiang Gong, Yue Zhang, et al . 2023. PromptBench: Towards Evaluating the Robustness of Large Language Models on Adversarial Prompts.arXiv preprint arXiv:2306.04528(2023)

    Pith/arXiv arXiv 2023

  67. [67]

    Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, and Kai Chen. 2024. ProSA: Assessing and Understanding the Prompt Sensitivity of LLMs. InFindings of the Association for Computational Linguistics: EMNLP 2024, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Association for Computational Linguistics, Miami, Florida, USA, 1950–19...

    doi:10.18653/v1/2024.findings-emnlp.108 2024

  68. [2025]

    InThe Thirteenth International Conference on Learning Representations

    Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=xoXn62FzD0