REVIEW 30 cited by
XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models
read the original abstract
The applications of LLM Agents are becoming increasingly complex and diverse, leading to a high demand for structured outputs that can be parsed into code, structured function calls, and embodied agent commands. These developments bring significant demands for structured generation in LLM inference. Context-free grammar is a flexible approach to enable structured generation via constrained decoding. However, executing context-free grammar requires going through several stack states over all tokens in vocabulary during runtime, bringing non-negligible overhead for structured generation. In this paper, we propose XGrammar, a flexible and efficient structure generation engine for large language models. XGrammar accelerates context-free grammar execution by dividing the vocabulary into context-independent tokens that can be prechecked and context-dependent tokens that need to be interpreted during runtime. We further build transformations to expand the grammar context and reduce the number of context-independent tokens. Additionally, we build an efficient persistent stack to accelerate the context-dependent token checks. Finally, we co-design the grammar engine with LLM inference engine to overlap grammar computation with GPU executions. Evaluation results show that XGrammar can achieve up to 100x speedup over existing solutions. Combined with an LLM inference engine, it can generate near-zero overhead structure generation in end-to-end low-LLM serving.
Forward citations
Cited by 30 Pith papers
-
Imprompt: A Language Framework for Prompt Programming
Prompt programs can be typed like programs and compiled in different ways; Imprompt formalizes two such compilers and tests them on tax-law reasoning and data-labeling tasks.
-
ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflows
ProtoMedAgent uses a privacy-aware agentic workflow with neuro-symbolic bottlenecks to achieve 91.2% faithfulness in clinical report generation, significantly outperforming standard RAG methods on a large patient cohort.
-
GraphInstruct: A Progressive Benchmark for Diagnosing Capability Gaps in LLM Graph Generation
GraphInstruct is a progressive benchmark with six complexity levels for LLM graph generation that identifies multi-constraint composition as the hardest point and shows a verification-guided iterative framework outper...
-
GraphInstruct: A Progressive Benchmark for Diagnosing Capability Gaps in LLM Graph Generation
GraphInstruct introduces a six-level progressive benchmark with 800 instructions and 1,582 references to diagnose LLM graph generation gaps, plus a verification-guided iterative prompting framework that improves performance.
-
The Structured Output Benchmark: A Multi-Source Benchmark for Evaluating Structured Output Quality in Large Language Models
SOB benchmark shows LLMs achieve near-perfect schema compliance but value accuracy of only 83% on text, 67% on images, and 24% on audio.
-
JSON Schema Inclusion through Refutational Normalization: Reconciling Efficiency and Completeness
Refutational normalization reconciles efficiency and completeness for JSON Schema inclusion checking.
-
Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees
Removing autoregressive decode from the decision-time critical path via pre-compiled guarded policy trees recovers contested GUI action windows when outcomes are enumerable in advance.
-
Grammar-Constrained Decoding Can Jailbreak LLMs into Generating Malicious Code
Grammar-constrained decoding enables a new jailbreak (CodeSpear) on LLMs for malicious code, countered by CodeShield which trains models to output harmless honeypot code under GCD while preserving refusals.
-
Transcoda: End-to-End Zero-Shot Optical Music Recognition via Data-Centric Synthetic Training
Transcoda achieves state-of-the-art zero-shot OMR with an 18.46% OMR-NED error rate on synthetic scores and 63.97% on historical Polish scans using a 59M model trained in 6 hours via synthetic data, kern normalization...
-
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
A practical evaluation protocol for AI pentesting agents that uses validated vulnerability discovery, LLM semantic matching, and bipartite scoring to assess performance in realistic, complex targets.
-
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
A compact OMR model trained only on normalized synthetic **kern data reaches 18.46% OMR-NED on Verovio scores and 63.97% on real Polish scans, outperforming larger baselines.
-
From Controlled to the Wild: Evaluation of Pentesting Agents for the Real-World
An evaluation protocol for AI pentesting agents that scores validated vulnerability discovery using LLM-based semantic matching and bipartite resolution.
-
NCO: A Versatile Plug-in for Handling Negative Constraints in Decoding
NCO enables efficient online pattern matching for negative hard and regex constraints in LLM decoding to prevent forbidden content without state explosion.
-
NEURA: A Unified and Retargetable Compilation Framework for Coarse-Grained Reconfigurable Architectures
NEURA flattens CGRA control flow into a pure predicated dataflow IR and reports 2.20× kernel and up to 2.71× application speedups over high-performance SOTA baselines.
-
The Format Tax
Structured-output instructions alone impose a large accuracy tax on open-weight LLMs; decoupling freeform reasoning from formatting recovers most of it, while recent closed models largely avoid the tax.
-
ATLAS-RTC: Closing the Loop on LLM Agent Output with Token-Level Runtime Control
ATLAS-RTC raises first-attempt success on structured LLM generation and tool calling by 20-37.8 points through closed-loop token-level interventions.
-
xGR: Efficient Generative Recommendation Serving at Scale
On real-world recommendation datasets, xGR sustains about 2.9–3.5× the throughput of vLLM/xLLM under a 200 ms P99 latency cap through GR-specific KV-cache, beam-search, and scheduling optimizations.
-
Orchestrating Dual-Boundaries: An Arithmetic Intensity Inspired Acceleration Framework for Diffusion Language Models
ODB-dLLM accelerates diffusion LLM inference by adaptively shortening responses and using a jump-share speculative decoding strategy, achieving 2.6-6.3x over Fast-dLLM.
-
RefineStat: Efficient Exploration for Probabilistic Program Synthesis
RefineStat improves small language model performance on probabilistic program synthesis by adding semantic constraint enforcement and diagnostic-aware refinement, producing syntactically and statistically reliable cod...
-
Empirical Study for Structured Output Control in LLMs for Software Engineering
Empirical benchmarks on four SE tasks show grammar-constrained decoding and TTMG eliminate most syntax errors in LLM outputs while structural and semantic errors persist and cascade in downstream tools.
-
ProtoMedAgent: Multimodal Clinical Interpretability via Privacy-Aware Agentic Workflows
ProtoMedAgent formalizes multimodal clinical reporting as iterative zero-gradient test-time optimization over a neuro-symbolic bottleneck with k-anonymity and ℓ-diversity privacy gate, reporting 91.2% faithfulness ver...
-
Agents of Diffusion: Enhancing Diffusion Language Models with Multi-Agent Reinforcement Learning for Structured Data Generation (Extended Version)
AoD pairs a frozen diffusion language model with two LLM agents that iteratively rewrite prompts from natural-language feedback, reporting better JSON diversity and validity, though the claimed RL mechanism and theore...
-
Large Databases Need Small, Open-Weight Language Models
Quantized open-weight LMs on consumer hardware match closed-source API accuracy for LM-enhanced relational operators while delivering 390x lower cost and 3.8x lower latency in the BlendSQL framework.
-
Ethical Hyper-Velocity (EHV): A Hardware-Rooted Zero-Trust Runtime Enforcement Architecture for Agentic AI Systems
EHV is a proposed JIT compiler architecture for agentic AI that relocates policy enforcement into the runtime pipeline and uses TLA+ to prove non-compliant actions are unreachable, reducing governance latency to O(1).
-
Ethical Hyper-Velocity (EHV): A Hardware-Rooted Zero-Trust Runtime Enforcement Architecture for Agentic AI Systems
EHV integrates GCD, causal graph CRDTs, TEE attestation, and bounded TLA+ verification to achieve O(1) runtime policy enforcement for agentic AI systems.
-
Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation
Compiled AI generates deterministic code artifacts from LLMs in a one-time compilation step, enabling reliable workflow execution with zero runtime tokens after break-even.
-
Precise Robot Command Understanding Using Grammar-Constrained Large Language Models
A fine-tuned LLM plus grammar canonicalizer and feedback loop yields higher valid robot-command rates on HuRIC than a fine-tuned LLM or grammar NLU alone.
-
A Survey on LLM-as-a-Judge
A survey on LLM-as-a-Judge that reviews reliability strategies, proposes evaluation methods, and introduces a novel benchmark for assessing such systems.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A comprehensive reference book organizing existing techniques for agentic AI systems across LLM substrate, reasoning, agent design patterns, inter-agent coordination, and production deployment.
-
The Hitchhiker's Guide to Agentic AI: From Foundations to Systems
A survey-style reference book mapping the full agentic-AI stack from transformer internals to production deployment, with no new research result.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.