REVIEW 11 cited by
SynCode: LLM Generation with Grammar Augmentation
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
SynCode: LLM Generation with Grammar Augmentation
read the original abstract
LLMs are widely used in complex AI applications. These applications underscore the need for LLM outputs to adhere to a specific format, for their integration with other components in the systems. Typically the format rules e.g., for data serialization formats such as JSON, YAML, or Code in Programming Language are expressed as context-free grammar (CFG). Due to the hallucinations and unreliability of LLMs, instructing LLMs to adhere to specified syntax becomes an increasingly important challenge. We present SynCode, a novel framework for efficient and general syntactical decoding with LLMs, to address this challenge. SynCode ensures soundness and completeness with respect to the CFG of a formal language, effectively retaining valid tokens while filtering out invalid ones. SynCode uses an offline-constructed, efficient lookup table, the DFA mask store, derived from the DFA of the language's grammar for efficient generation. SynCode seamlessly integrates with any language defined by CFG, as evidenced by experiments focusing on generating JSON, Python, and Go outputs. Our experiments evaluating the effectiveness of SynCode for JSON generation demonstrate that SynCode eliminates all syntax errors and significantly outperforms state-of-the-art baselines. Furthermore, our results underscore how SynCode significantly reduces 96.07% of syntax errors in generated Python and Go code, showcasing its substantial impact on enhancing syntactical precision in LLM generation. Our code is available at https://github.com/uiuc-focal-lab/syncode
Forward citations
Cited by 11 Pith papers
-
TruncProof: A Guardrail for LLM-based JSON Generation under Token-Length Constraints
TruncProof lets LLMs generate syntactically valid JSON within strict token limits by approximating completion token counts via LL(1) parser lookahead.
-
When Correct Isn't Usable: Improving Structured Output Reliability in Small Language Models
AloLab, an iterative meta-agent prompt optimizer, raises structured output accuracy for 7-9B models from 0% to 84-87% on GSM8K while preserving near-native inference speed.
-
TreeCoder: Systematic Exploration and Optimisation of Decoding and Constraints for LLM Code Generation
TreeCoder improves LLM code generation accuracy by representing decoding as an optimizable tree search over programs with first-class constraints for syntax, style, and execution, outperforming baselines on MBPP and S...
-
RefineStat: Efficient Exploration for Probabilistic Program Synthesis
RefineStat improves small language model performance on probabilistic program synthesis by adding semantic constraint enforcement and diagnostic-aware refinement, producing syntactically and statistically reliable cod...
-
Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking
Across 140k generations, wrapper-only formatting changes accuracy by over 30× across models, largely via parseability failures, so single-wrapper scores are statistically fragile.
-
LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition
Reference-free LLM judge scores failed to select better table-extraction outputs over eight regeneration iterations on FinTabNet and OmniDocBench; keeping the first output was safest.
-
Empirical Study for Structured Output Control in LLMs for Software Engineering
Empirical benchmarks on four SE tasks show grammar-constrained decoding and TTMG eliminate most syntax errors in LLM outputs while structural and semantic errors persist and cascade in downstream tools.
-
TruncProof: A Guardrail for LLM-based JSON Generation under Token-Length Constraints
TruncProof is a grammar-constrained decoding method that enforces token-length limits on LLM-generated JSON by using LL(1) parser lookahead to estimate remaining tokens needed for syntactic validity.
-
AdaDec: A Uncertainty-Guided Lookahead Decoding Framework for LLM-Based Code Generation
AdaDec improves Pass@1 accuracy of LLM code generation by up to 20.9% over greedy decoding by triggering lookahead reranking only at high-uncertainty steps on HumanEval+, MBPP+, and DevEval.
-
Compiled AI: Deterministic Code Generation for LLM-Based Workflow Automation
Compiled AI generates deterministic code artifacts from LLMs in a one-time compilation step, enabling reliable workflow execution with zero runtime tokens after break-even.
-
Case Study: Fine-tuning Small Language Models for Accurate and Private CWE Detection in Python Code
Fine-tuning codegen-mono on 500 synthetic-plus-reviewed Python examples produces 99% accuracy, 98.08% precision, 100% recall and 99.04% F1 for CWE detection.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.