Pith. sign in

REVIEW 20 cited by

SynCode: LLM Generation with Grammar Augmentation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.01632 v4 pith:SVLX5XHI submitted 2024-03-03 cs.LG cs.FLcs.PLcs.SE

Shubham Ugare , Tarun Suresh , Hangoo Kang , Sasa Misailovic , Gagandeep Singh This is my paper · ORCID
classification cs.LGcs.FLcs.PLcs.SE
keywords syncodegenerationlanguagellmscodeefficientgrammarjson
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

LLMs are widely used in complex AI applications. These applications underscore the need for LLM outputs to adhere to a specific format, for their integration with other components in the systems. Typically the format rules e.g., for data serialization formats such as JSON, YAML, or Code in Programming Language are expressed as context-free grammar (CFG). Due to the hallucinations and unreliability of LLMs, instructing LLMs to adhere to specified syntax becomes an increasingly important challenge. We present SynCode, a novel framework for efficient and general syntactical decoding with LLMs, to address this challenge. SynCode ensures soundness and completeness with respect to the CFG of a formal language, effectively retaining valid tokens while filtering out invalid ones. SynCode uses an offline-constructed, efficient lookup table, the DFA mask store, derived from the DFA of the language's grammar for efficient generation. SynCode seamlessly integrates with any language defined by CFG, as evidenced by experiments focusing on generating JSON, Python, and Go outputs. Our experiments evaluating the effectiveness of SynCode for JSON generation demonstrate that SynCode eliminates all syntax errors and significantly outperforms state-of-the-art baselines. Furthermore, our results underscore how SynCode significantly reduces 96.07% of syntax errors in generated Python and Go code, showcasing its substantial impact on enhancing syntactical precision in LLM generation. Our code is available at https://github.com/uiuc-focal-lab/syncode

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. DINGO: Constrained Inference for Diffusion LLMs

    cs.LG 2025-05 conditional novelty 7.0 of 10

    DINGO is a dynamic programming decoder over a token-level DFA that makes diffusion LLM outputs obey regular expressions while maximizing the product of per-position probabilities.

  2. Type-Driven Prompt Programming: From Typed Interfaces to a Calculus of Constraints

    cs.PL 2025-08 conditional novelty 6.0 of 10

    The paper proposes a not-yet-complete dependently typed calculus for prompt programming with probabilistic refinements, and identifies gaps in constraint expressiveness and optimization algorithms.

  3. Syntax Repair as Language Intersection

    cs.FL 2025-07 conditional novelty 6.0 of 10

    Bounded syntax repair is formalized as the intersection of a context-free grammar with an acyclic Levenshtein automaton, and the paper shows this intersection can be built, enumerated, and reranked efficiently.

  4. Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlo

    cs.CL 2025-04 conditional novelty 6.0 of 10

    Sequential Monte Carlo control of LLM generation, which checks partial outputs and resamples promising candidates, lets small open models beat much larger closed models on constrained code, SQL, goal, and molecule generation.

  5. Memory Offloading for Large Language Model Inference with Latency SLO Guarantees

    cs.DC 2025-02 conditional novelty 6.0 of 10

    Select-N meets LLM latency SLOs by choosing and continuously adjusting an offloading interval that controls how much model state lives in CPU memory, increasing host memory usage and throughput versus prior offloading...

  6. Optimizing Datasets for Code Summarization: Is Code-Comment Coherence Enough?

    cs.SE 2025-02 conditional novelty 6.0 of 10

    Filtering code summarization training data by code-comment coherence gives no better results than random selection, and halving the training set does not hurt performance.

  7. Flexible and Efficient Grammar-Constrained Decoding

    cs.CL 2025-02 conditional novelty 6.0 of 10

    GREATGRAMMA precomputes only the terminal sequences an LLM tokenizer can actually produce, yielding 17.71x faster offline preprocessing than Syncode and 5 to 32 ms online masking per token.

  8. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models

    cs.CL 2024-11 conditional novelty 6.0 of 10

    XGrammar accelerates context-free-grammar constrained LLM decoding by caching context-independent token masks, checking only context-dependent tokens at runtime, and overlapping mask generation with GPU inference.

  9. Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

    cs.AI 2026-05 conditional novelty 5.5 of 10

    Across 140k generations, wrapper-only formatting changes accuracy by over 30× across models, largely via parseability failures, so single-wrapper scores are statistically fragile.

  10. LLM-as-a-Judge Scores Are Unreliable Optimization Signals in Closed-Loop Table Recognition

    cs.CL 2026-07 conditional novelty 5.0 of 10

    Reference-free LLM judge scores failed to select better table-extraction outputs over eight regeneration iterations on FinTabNet and OmniDocBench; keeping the first output was safest.

  11. Assessing Large Language Models in Comprehending and Verifying Concurrent Programs across Memory Models

    cs.SE 2025-01 conditional novelty 5.0 of 10

    Five LLMs, including GPT-4, handle many simple concurrency checks but cannot reliably verify small programs under TSO and PSO relaxed memory models.

  12. FREYR: A Framework for Recognizing and Executing Your Requests

    cs.SE 2025-01 conditional novelty 5.0 of 10

    FREYR, a modular pipeline that separates intent detection, parameter generation, and summarization, achieves higher task completion than Ollama's native tool calling on the LLMaker benchmark.

  13. Formal Mathematical Reasoning: A New Frontier in AI

    cs.AI 2024-12 conditional novelty 5.0 of 10

    Machine-checkable formal proof should become the backbone of AI mathematics, and a five-task, five-level capability roadmap can measure progress toward that goal.

  14. From Words to Workflows: Automating Business Processes

    cs.AI 2024-12 conditional novelty 5.0 of 10

    Text2Workflow is a multi-prompt LLM system with human feedback that generates JSON workflows from natural language, scoring 71.3% average semantic accuracy on the authors' 60-request Process2JSON dataset, versus 64.2%...

  15. Syntactic Control of Language Models by Posterior Inference

    cs.CL 2025-06 conditional novelty 4.0 of 10

    SMC sampling with a tetratagger shaping function and a POS-bigram proposal lifts constituency-tree match from 12-35% to about 93% F1 on GPT2-large and Llama3-8B.

  16. Constrained Sampling for Language Models Should Be Easy: An MCMC Perspective

    cs.AI 2025-06 conditional novelty 4.0 of 10

    A grammar-aligned MCMC sampler with prefix-resampling proposals and Metropolis-Hastings acceptance converges to the constrained language model distribution and improves fuzzing seed coverage.

  17. Multiple Abstraction Level Retrieve Augment Generation

    cs.CL 2025-01 conditional novelty 4.0 of 10

    MAL-RAG retrieves document, section, paragraph, and multi-sentence chunks together and claims a 25.7% improvement in AI-judged answer correctness on glycoscience questions over single-level RAG.

  18. CoopetitiveV: Leveraging LLM-powered Coopetitive Multi-Agent Prompting for High-quality Verilog Generation

    cs.LG 2024-12 reject novelty 4.0 of 10

    A coopetitive multi-agent LLM framework, with a researcher, a prosecutor critic, and two revisers, pushes Verilog code generation to near-perfect pass rates on standard benchmarks.

  19. Using a Feedback Loop for LLM-based Infrastructure as Code Generation

    cs.SE 2024-11 conditional novelty 4.0 of 10

    Feeding cfn-lint errors back to an LLM reduces CloudFormation template errors for a few rounds, but the improvement plateaus by roughly the fifth iteration.

  20. Hallucination by Code Generation LLMs: Taxonomy, Benchmarks, Mitigation, and Challenges

    cs.SE 2025-04 conditional novelty 3.0 of 10

    This survey organizes code-LLM hallucinations into four types, reviews 52 related papers, and identifies open challenges in detection and mitigation.

Pith tools