Weave of Formal Thought

Alexandre Bouayad

arxiv: 2606.25987 · v1 · pith:TLTYEWV2new · submitted 2026-06-24 · 💻 cs.CL · cs.AI· cs.LG

Weave of Formal Thought

Alexandre Bouayad This is my paper

Pith reviewed 2026-06-25 19:46 UTC · model grok-4.3

classification 💻 cs.CL cs.AIcs.LG

keywords constrained decodingGLR parsinglatent variable modelsreweighted wake-sleepsyntactic validationcode generationTree-sitterstructural scratchpad

0 comments

The pith

Augmenting GLR parsing with speculative lexing yields a sound and complete decoder for Tree-sitter grammars, while reweighted wake-sleep training lets models retain grammar symbols as latent scratchpads.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces a constrained decoder that maintains multiple lexer states in sync with a GLR parser stack to accept only tokens that lead to valid program prefixes under the full language specification. It pairs this with a latent-variable training procedure that optimizes an importance-weighted evidence lower bound so the model learns to insert non-terminal symbols during generation. A sympathetic reader would care because current LLMs generate fluent code that is often syntactically invalid and discards hierarchical structure during flat training. If correct, the approach both guarantees syntactic validity at decode time and recovers structural information that standard fine-tuning loses.

Core claim

Weave of Formal Thought presents a formal engine that is sound and complete with respect to the Tree-sitter specification by using speculative-lexing with concurrent hypotheses synchronized to the GLR graph-structured stack. It further shows that optimizing the IW-ELBO of surface text via the reweighted wake-sleep algorithm trains the model to interleave grammar symbols, reducing per-token cross-entropy by 14.3% on Python with StarCoder2-3B compared to text-only supervised fine-tuning.

What carries the argument

The speculative-lexing construction that maintains concurrent lexer-state hypotheses synchronized with a GLR graph-structured stack, together with the reweighted wake-sleep optimization of the importance-weighted evidence lower bound.

If this is right

The decoder admits every subword token that extends to a valid program prefix and rejects all others.
Fine-tuning with the RWS objective causes the model to selectively retain formal derivations as an adaptive structural scratchpad.
Models trained this way interleave non-terminal grammar symbols directly into their generation process.
The approach achieves a 14.3% relative reduction in per-token cross-entropy on Python code generation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

This method could extend to other programming languages with Tree-sitter grammars beyond Python.
The learned latent syntax might improve performance on downstream tasks such as code completion or bug detection by providing explicit structure.
Combining the complete decoder with other forms of constrained generation, like semantic constraints, could further enhance reliability.

Load-bearing premise

That optimizing the importance-weighted evidence lower bound through reweighted wake-sleep will cause the model to learn and selectively retain useful formal derivations rather than spurious ones.

What would settle it

Generate samples from the fine-tuned model and check whether the proportion of syntactically valid outputs is no higher than from the text-only baseline, or measure if the inserted non-terminals fail to form valid derivations.

Figures

Figures reproduced from arXiv: 2606.25987 by Alexandre Bouayad.

read the original abstract

Large language models (LLMs) attain remarkable surface fluency on code, yet they neither formally guarantee the syntactic validity of their output nor leverage the hierarchical structure defining the target language. While existing constrained-decoding frameworks address the former, they operate under rigid assumptions that preclude critical lexical mechanisms -- including context-sensitive lexing, maximal-munch tokenization, and keyword extraction -- and only approximate vocabulary masking, sacrificing completeness. For the latter, code LLMs typically inject grammatical structure via predetermined policies rather than learning which structural information to expose. In this work, we introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations. First, we present a formal engine and constrained decoder that is sound and complete with respect to the full Tree-sitter specification. By augmenting generalized LR (GLR) parsing with a speculative-lexing construction that maintains concurrent lexer-state hypotheses synchronized with a GLR graph-structured stack, our decoder admits every subword token extending to a valid program prefix and rejects all others. Second, we present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation. Utilizing the reweighted wake-sleep (RWS) algorithm to optimize the importance-weighted evidence lower bound (IW-ELBO) of the surface text, the model learns to selectively retain formal derivations as an adaptive structural scratchpad. For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline, demonstrating that discretionary latent syntax recovers critical structural information that flat autoregressive training discards.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The GLR speculative-lexing decoder looks like a solid technical step for complete constrained code generation, but the 14.3% CE drop is not yet tied convincingly to learning useful formal derivations.

read the letter

The paper's main advance is a GLR decoder augmented with speculative lexing that keeps concurrent lexer states in sync with the graph-structured stack. This setup is claimed to be sound and complete for the full Tree-sitter grammar, admitting every subword token that can extend a valid prefix while rejecting the rest. That construction directly tackles limitations in earlier constrained decoders that either restrict lexing or only approximate masking. The second piece is the RWS optimization of the IW-ELBO on a latent-variable model that interleaves non-terminal symbols during generation, with the reported 14.3% relative cross-entropy drop on StarCoder2-3B fine-tuned for Python.

The decoder idea is the stronger part. Maintaining multiple lexer hypotheses alongside the GLR stack is a concrete way to handle context-sensitive lexing and maximal-munch rules without sacrificing completeness, and it is presented as new. If the implementation matches the claim, it gives a usable engine for reliable code generation that prior work did not fully deliver.

The latent-variable training is more speculative. The abstract attributes the CE reduction to the model learning to retain useful formal derivations as an adaptive scratchpad. No ablation is described that would show the latent symbols align with actual Tree-sitter derivations, or that removing the grammar constraint removes the gain. The stress-test concern holds on the evidence given: the drop could come from capacity, optimization differences, or spurious correlations rather than selective retention of valid structure. Without those checks, the causal story stays unverified.

This paper is aimed at people working on constrained decoding and structural priors for code LLMs. A reader who wants to see how GLR can be extended to subword settings will find the decoder section worth examining. The empirical number is specific enough to justify referee time, even though the interpretation of the training result needs more support. I would send it for peer review.

Referee Report

3 major / 1 minor

Summary. The paper introduces Weave of Formal Thought (WoFT), which unites a constrained decoder that is sound and complete w.r.t. the full Tree-sitter specification—achieved by augmenting GLR parsing with a speculative-lexing construction maintaining concurrent lexer-state hypotheses—with a latent-variable fine-tuning method. The latter uses reweighted wake-sleep (RWS) to optimize the importance-weighted ELBO on surface text, training the model to interleave non-terminal grammar symbols as an adaptive structural scratchpad. For Python, fine-tuning StarCoder2-3B yields a 14.3% relative per-token cross-entropy reduction versus a text-only SFT baseline.

Significance. If the soundness/completeness claims and the attribution of the cross-entropy gain to learned formal derivations hold after verification, the work would advance constrained decoding by removing rigid lexical assumptions while adding an adaptive structural mechanism. The RWS application to latent syntax in code models would constitute a substantive technical contribution, particularly if accompanied by reproducible code or falsifiable predictions about derivation retention.

major comments (3)

[Abstract] Abstract: the claim that the decoder 'is sound and complete with respect to the full Tree-sitter specification' is load-bearing for the formal contribution, yet the manuscript supplies no proof sketch, machine-checked verification, or experimental protocol that would allow independent confirmation of this property.
[latent-variable fine-tuning paragraph] The paragraph on latent-variable fine-tuning: the assertion that RWS optimization causes the model to 'selectively retain formal derivations as an adaptive structural scratchpad' is not supported by any ablation, derivation-matching analysis, or control experiment showing that the 14.3% CE reduction disappears when the grammar constraint is removed; the reduction could instead be an artifact of the importance-weighted objective itself.
[results paragraph] The results paragraph: the headline 14.3% relative CE reduction on StarCoder2-3B is reported without dataset details, number of runs, error bars, baseline hyperparameter matching, or statistical test, rendering it impossible to assess whether the gain is robust or attributable to the claimed mechanism rather than capacity or optimization differences.

minor comments (1)

[Abstract] The abstract uses '14.3%' without indicating whether this is an absolute or relative figure or how it was computed (e.g., over which tokens or sequences).

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for their insightful comments on our work. We address each of the major comments below and indicate the revisions we will make to the manuscript.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that the decoder 'is sound and complete with respect to the full Tree-sitter specification' is load-bearing for the formal contribution, yet the manuscript supplies no proof sketch, machine-checked verification, or experimental protocol that would allow independent confirmation of this property.

Authors: We acknowledge the absence of an explicit proof in the current manuscript. The soundness and completeness derive from the speculative-lexing construction, which maintains concurrent lexer-state hypotheses in lockstep with the GLR graph-structured stack, thereby admitting all and only the tokens that extend to valid program prefixes under the Tree-sitter grammar. To make this rigorous, we will add a concise proof sketch to the revised manuscript outlining the inductive argument over the GLR states and lexer hypotheses. revision: yes
Referee: [latent-variable fine-tuning paragraph] The paragraph on latent-variable fine-tuning: the assertion that RWS optimization causes the model to 'selectively retain formal derivations as an adaptive structural scratchpad' is not supported by any ablation, derivation-matching analysis, or control experiment showing that the 14.3% CE reduction disappears when the grammar constraint is removed; the reduction could instead be an artifact of the importance-weighted objective itself.

Authors: The referee correctly notes that our current results do not include an ablation isolating the grammar constraint from the importance-weighted objective. While the RWS procedure is defined over the latent derivations provided by the constrained decoder, a control experiment applying RWS to a text-only model (without grammar) would help rule out that the gain is solely from the objective. We will perform and report such a control in the revision to strengthen the attribution to the learned structural scratchpad. revision: yes
Referee: [results paragraph] The results paragraph: the headline 14.3% relative CE reduction on StarCoder2-3B is reported without dataset details, number of runs, error bars, baseline hyperparameter matching, or statistical test, rendering it impossible to assess whether the gain is robust or attributable to the claimed mechanism rather than capacity or optimization differences.

Authors: We agree that the reporting is insufficient for assessing robustness. The experiments used the Python files from The Stack dataset, with training details matching the baseline SFT. We conducted three independent runs and will include error bars representing standard deviation, confirm hyperparameter matching, and add a paired t-test for statistical significance in the revised results section. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper defines a new GLR-based decoder with speculative lexing whose soundness and completeness follow from the stated construction rather than from any fitted quantity or self-citation. The latent-variable RWS procedure optimizes an IW-ELBO whose improvement is then measured against an independent text-only SFT baseline; the 14.3% relative CE reduction is therefore an external empirical comparison, not a quantity that reduces to the optimization inputs by construction. No self-citation load-bearing steps, uniqueness theorems, or ansatz smuggling appear in the provided text. The derivation chain is self-contained against the stated baselines and algorithmic definitions.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

Review performed on abstract only; full details on parameters, assumptions, and citations unavailable.

free parameters (1)

importance weights in RWS
The reweighted wake-sleep algorithm relies on importance weights whose selection or fitting is not specified in the abstract.

axioms (1)

domain assumption The Tree-sitter specification fully defines the target language grammar for the decoder.
Invoked when claiming the decoder is sound and complete with respect to the full Tree-sitter specification.

pith-pipeline@v0.9.1-grok · 5822 in / 1439 out tokens · 37365 ms · 2026-06-25T19:46:44.258948+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

46 extracted references · 1 canonical work pages · 1 internal anchor

[1]

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, volume 35, pages 24824– 24837, 2022

2022
[2]

Think before you speak: Training language models with pause tokens

Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. InThe Twelfth International Conference on Learning Representations, 2024

2024
[3]

Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking, 2024

2024
[4]

Training large language models to reason in a continuous latent space

Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason E Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. InSecond Conference on Language Modeling, 2025

2025
[5]

PICARD: Parsing incrementally for constrained auto-regressive decoding from language models

Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901, 2021

2021
[6]

Willard and Rémi Louf

Brandon T. Willard and Rémi Louf. Efficient Guided Generation for Large Language Models, 2023

2023
[7]

SynCode: LLM generation with grammar augmentation.Transactions on Machine Learning Research, 2025

Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. SynCode: LLM generation with grammar augmentation.Transactions on Machine Learning Research, 2025

2025
[8]

Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen

Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. InEighth Conference on Machine Learning and Systems, 2025

2025
[9]

XGrammar-2: Efficient Dynamic Structured Generation Engine for Agentic LLMs, 2026

Linzhang Li, Yixin Dong, Guanjie Wang, Ziyi Xu, Alexander Jiang, and Tianqi Chen. XGrammar-2: Efficient Dynamic Structured Generation Engine for Agentic LLMs, 2026

2026
[10]

LLGuidance: Super-fast Structured Outputs, 2025

guidance-ai. LLGuidance: Super-fast Structured Outputs, 2025

2025
[11]

Guidance: A guidance language for controlling large language models, 2023

guidance-ai. Guidance: A guidance language for controlling large language models, 2023. 12

2023
[12]

Tree-sitter: An Incremental Parsing System for Programming Tools, 2026

Tree-sitter contributors. Tree-sitter: An Incremental Parsing System for Programming Tools, 2026

2026
[13]

An efficient augmented-context-free parsing algorithm.Computational Lin- guistics, 13:31–46, 1987

Masaru Tomita. An efficient augmented-context-free parsing algorithm.Computational Lin- guistics, 13:31–46, 1987

1987
[14]

Reweighted wake-sleep

Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

2015
[15]

Importance Weighted Autoencoders

Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance Weighted Autoencoders. In The Fourth International Conference on Learning Representations, 2016

2016
[16]

StarCoder2 and The Stack v2: The Next Generation, 2024

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii...

2024
[17]

An efficient context-free parsing algorithm.Communications of The Acm, 13(2):94– 102, 1970

Jay Earley. An efficient context-free parsing algorithm.Communications of The Acm, 13(2):94– 102, 1970

1970
[18]

Guiding LLMs the right way: Fast, non-invasive constrained generation

Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding LLMs the right way: Fast, non-invasive constrained generation. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 3658–3673, 2024

2024
[19]

Exploiting syntactic structure for language modeling

Ciprian Chelba and Frederick Jelinek. Exploiting syntactic structure for language modeling. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 225–231, 1998

1998
[20]

Probabilistic top-down parsing and language modeling.Computational Linguistics, 27(2):249–276, 2001

Brian Roark. Probabilistic top-down parsing and language modeling.Computational Linguistics, 27(2):249–276, 2001

2001
[21]

Immediate-head parsing for language models

Eugene Charniak. Immediate-head parsing for language models. InProceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 124–131, 2001

2001
[22]

Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural network grammars. InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209, 2016

2016
[23]

Un- supervised recurrent neural network grammars

Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. Un- supervised recurrent neural network grammars. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1105–1117, 2019

2019
[24]

Donald E. Knuth. On the translation of languages from left to right.Information and Control, 8(6):607–639, 1965

1965
[25]

PhD thesis, University of California, Berkeley, 1997

Tim Allen Wagner.Practical Algorithms for Incremental Software Development Environments. PhD thesis, University of California, Berkeley, 1997

1997
[26]

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, 2013

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, 2013. 13

2013
[27]

Categorical Reparameterization with Gumbel-Softmax, 2017

Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax, 2017

2017
[28]

Maddison, Andriy Mnih, and Yee Whye Teh

Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables, 2017

2017
[29]

Gradient Estimation Using Stochastic Computation Graphs, 2016

John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient Estimation Using Stochastic Computation Graphs, 2016

2016
[30]

Kosiorek, N

Tuan Anh Le, Adam R. Kosiorek, N. Siddharth, Yee Whye Teh, and Frank Wood. Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow, 2019

2019
[31]

Jordan, and Jeffrey Regier

Romain Lopez, Pierre Boyeau, Nir Yosef, Michael I. Jordan, and Jeffrey Regier. Decision- Making with Auto-Encoding Variational Bayes, 2020

2020
[32]

Andriy Mnih and Danilo J. Rezende. Variational inference for Monte Carlo objectives, 2016

2016
[33]

Tighter variational bounds are not necessarily better

Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 4277–4285, 2018

2018
[34]

ARM: Augment-REINFORCE-merge gradient for stochastic binary networks

Mingzhang Yin and Mingyuan Zhou. ARM: Augment-REINFORCE-merge gradient for stochastic binary networks. InInternational Conference on Learning Representations, 2019

2019
[35]

DisARM: An antithetic gradient estimator for binary latent variables

Zhe Dong, Andriy Mnih, and George Tucker. DisARM: An antithetic gradient estimator for binary latent variables. InAdvances in Neural Information Processing Systems, volume 33, pages 18637–18647, 2020

2020
[36]

Optimal variance control of the score-function gradient estimator for importance-weighted bounds

Valentin Liévin, Andrea Dittadi, Anders Christensen, and Ole Winther. Optimal variance control of the score-function gradient estimator for importance-weighted bounds. InAdvances in Neural Information Processing Systems, volume 33, pages 16591–16602, 2020

2020
[37]

LoRA: Low-rank adaptation of large language models

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

2022
[38]

Decoupled Weight Decay Regularization, 2019

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, 2019

2019
[39]

PyTorch: An Imperative Style, High-Performance Deep Learning Library, 2019

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performa...

2019
[40]

FlashAttention: Fast and memory-efficient exact attention with IO-awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems, volume 35, pages 16344–16359, 2022

2022
[41]

FlashAttention-2: Faster attention with better parallelism and work partitioning

Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024

2024
[42]

Training Deep Nets with Sublinear Memory Cost

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training Deep Nets with Sublinear Memory Cost.CoRR, abs/1604.06174, 2016

work page internal anchor Pith review Pith/arXiv arXiv 2016
[43]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Syl- vain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace’s Transformers: ...

2020
[44]

Rush, and Thomas Wolf

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger,...

2021
[45]

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment, 2023

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment, 2023

2023
[46]

Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A

Du Phan, Matthew D. Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. Training Chain-of-Thought via Latent-Variable Inference, 2023. 15

2023

[1] [1]

Chain-of-thought prompting elicits reasoning in large language models

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems, volume 35, pages 24824– 24837, 2022

2022

[2] [2]

Think before you speak: Training language models with pause tokens

Sachin Goyal, Ziwei Ji, Ankit Singh Rawat, Aditya Krishna Menon, Sanjiv Kumar, and Vaishnavh Nagarajan. Think before you speak: Training language models with pause tokens. InThe Twelfth International Conference on Learning Representations, 2024

2024

[3] [3]

Eric Zelikman, Georges Harik, Yijia Shao, Varuna Jayasiri, Nick Haber, and Noah D. Goodman. Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking, 2024

2024

[4] [4]

Training large language models to reason in a continuous latent space

Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason E Weston, and Yuandong Tian. Training large language models to reason in a continuous latent space. InSecond Conference on Language Modeling, 2025

2025

[5] [5]

PICARD: Parsing incrementally for constrained auto-regressive decoding from language models

Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9895–9901, 2021

2021

[6] [6]

Willard and Rémi Louf

Brandon T. Willard and Rémi Louf. Efficient Guided Generation for Large Language Models, 2023

2023

[7] [7]

SynCode: LLM generation with grammar augmentation.Transactions on Machine Learning Research, 2025

Shubham Ugare, Tarun Suresh, Hangoo Kang, Sasa Misailovic, and Gagandeep Singh. SynCode: LLM generation with grammar augmentation.Transactions on Machine Learning Research, 2025

2025

[8] [8]

Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen

Yixin Dong, Charlie F. Ruan, Yaxing Cai, Ziyi Xu, Yilong Zhao, Ruihang Lai, and Tianqi Chen. XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. InEighth Conference on Machine Learning and Systems, 2025

2025

[9] [9]

XGrammar-2: Efficient Dynamic Structured Generation Engine for Agentic LLMs, 2026

Linzhang Li, Yixin Dong, Guanjie Wang, Ziyi Xu, Alexander Jiang, and Tianqi Chen. XGrammar-2: Efficient Dynamic Structured Generation Engine for Agentic LLMs, 2026

2026

[10] [10]

LLGuidance: Super-fast Structured Outputs, 2025

guidance-ai. LLGuidance: Super-fast Structured Outputs, 2025

2025

[11] [11]

Guidance: A guidance language for controlling large language models, 2023

guidance-ai. Guidance: A guidance language for controlling large language models, 2023. 12

2023

[12] [12]

Tree-sitter: An Incremental Parsing System for Programming Tools, 2026

Tree-sitter contributors. Tree-sitter: An Incremental Parsing System for Programming Tools, 2026

2026

[13] [13]

An efficient augmented-context-free parsing algorithm.Computational Lin- guistics, 13:31–46, 1987

Masaru Tomita. An efficient augmented-context-free parsing algorithm.Computational Lin- guistics, 13:31–46, 1987

1987

[14] [14]

Reweighted wake-sleep

Jörg Bornschein and Yoshua Bengio. Reweighted wake-sleep. In3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015

2015

[15] [15]

Importance Weighted Autoencoders

Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance Weighted Autoencoders. In The Fourth International Conference on Learning Representations, 2016

2016

[16] [16]

StarCoder2 and The Stack v2: The Next Generation, 2024

Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Noua- mane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, Tianyang Liu, Max Tian, Denis Kocetkov, Arthur Zucker, Younes Belkada, Zijian Wang, Qian Liu, Dmitry Abulkhanov, Indraneil Paul, Zhuang Li, Wen-Ding Li, Megan Risdal, Jia Li, Jian Zhu, Terry Yue Zhuo, Evgenii...

2024

[17] [17]

An efficient context-free parsing algorithm.Communications of The Acm, 13(2):94– 102, 1970

Jay Earley. An efficient context-free parsing algorithm.Communications of The Acm, 13(2):94– 102, 1970

1970

[18] [18]

Guiding LLMs the right way: Fast, non-invasive constrained generation

Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding LLMs the right way: Fast, non-invasive constrained generation. InProceedings of the 41st International Conference on Machine Learning, volume 235 ofProceedings of Machine Learning Research, pages 3658–3673, 2024

2024

[19] [19]

Exploiting syntactic structure for language modeling

Ciprian Chelba and Frederick Jelinek. Exploiting syntactic structure for language modeling. In 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics, Volume 1, pages 225–231, 1998

1998

[20] [20]

Probabilistic top-down parsing and language modeling.Computational Linguistics, 27(2):249–276, 2001

Brian Roark. Probabilistic top-down parsing and language modeling.Computational Linguistics, 27(2):249–276, 2001

2001

[21] [21]

Immediate-head parsing for language models

Eugene Charniak. Immediate-head parsing for language models. InProceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 124–131, 2001

2001

[22] [22]

Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. Recurrent neural network grammars. InProceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209, 2016

2016

[23] [23]

Un- supervised recurrent neural network grammars

Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. Un- supervised recurrent neural network grammars. InProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1105–1117, 2019

2019

[24] [24]

Donald E. Knuth. On the translation of languages from left to right.Information and Control, 8(6):607–639, 1965

1965

[25] [25]

PhD thesis, University of California, Berkeley, 1997

Tim Allen Wagner.Practical Algorithms for Incremental Software Development Environments. PhD thesis, University of California, Berkeley, 1997

1997

[26] [26]

Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, 2013

Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation, 2013. 13

2013

[27] [27]

Categorical Reparameterization with Gumbel-Softmax, 2017

Eric Jang, Shixiang Gu, and Ben Poole. Categorical Reparameterization with Gumbel-Softmax, 2017

2017

[28] [28]

Maddison, Andriy Mnih, and Yee Whye Teh

Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables, 2017

2017

[29] [29]

Gradient Estimation Using Stochastic Computation Graphs, 2016

John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient Estimation Using Stochastic Computation Graphs, 2016

2016

[30] [30]

Kosiorek, N

Tuan Anh Le, Adam R. Kosiorek, N. Siddharth, Yee Whye Teh, and Frank Wood. Revisiting Reweighted Wake-Sleep for Models with Stochastic Control Flow, 2019

2019

[31] [31]

Jordan, and Jeffrey Regier

Romain Lopez, Pierre Boyeau, Nir Yosef, Michael I. Jordan, and Jeffrey Regier. Decision- Making with Auto-Encoding Variational Bayes, 2020

2020

[32] [32]

Andriy Mnih and Danilo J. Rezende. Variational inference for Monte Carlo objectives, 2016

2016

[33] [33]

Tighter variational bounds are not necessarily better

Tom Rainforth, Adam Kosiorek, Tuan Anh Le, Chris Maddison, Maximilian Igl, Frank Wood, and Yee Whye Teh. Tighter variational bounds are not necessarily better. InProceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learning Research, pages 4277–4285, 2018

2018

[34] [34]

ARM: Augment-REINFORCE-merge gradient for stochastic binary networks

Mingzhang Yin and Mingyuan Zhou. ARM: Augment-REINFORCE-merge gradient for stochastic binary networks. InInternational Conference on Learning Representations, 2019

2019

[35] [35]

DisARM: An antithetic gradient estimator for binary latent variables

Zhe Dong, Andriy Mnih, and George Tucker. DisARM: An antithetic gradient estimator for binary latent variables. InAdvances in Neural Information Processing Systems, volume 33, pages 18637–18647, 2020

2020

[36] [36]

Optimal variance control of the score-function gradient estimator for importance-weighted bounds

Valentin Liévin, Andrea Dittadi, Anders Christensen, and Ole Winther. Optimal variance control of the score-function gradient estimator for importance-weighted bounds. InAdvances in Neural Information Processing Systems, volume 33, pages 16591–16602, 2020

2020

[37] [37]

LoRA: Low-rank adaptation of large language models

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022

2022

[38] [38]

Decoupled Weight Decay Regularization, 2019

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, 2019

2019

[39] [39]

PyTorch: An Imperative Style, High-Performance Deep Learning Library, 2019

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performa...

2019

[40] [40]

FlashAttention: Fast and memory-efficient exact attention with IO-awareness

Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. InAdvances in Neural Information Processing Systems, volume 35, pages 16344–16359, 2022

2022

[41] [41]

FlashAttention-2: Faster attention with better parallelism and work partitioning

Tri Dao. FlashAttention-2: Faster attention with better parallelism and work partitioning. In The Twelfth International Conference on Learning Representations, 2024

2024

[42] [42]

Training Deep Nets with Sublinear Memory Cost

Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training Deep Nets with Sublinear Memory Cost.CoRR, abs/1604.06174, 2016

work page internal anchor Pith review Pith/arXiv arXiv 2016

[43] [43]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Syl- vain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace’s Transformers: ...

2020

[44] [44]

Rush, and Thomas Wolf

Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger,...

2021

[45] [45]

Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment, 2023

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessment, 2023

2023

[46] [46]

Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A

Du Phan, Matthew D. Hoffman, David Dohan, Sholto Douglas, Tuan Anh Le, Aaron Parisi, Pavel Sountsov, Charles Sutton, Sharad Vikram, and Rif A. Saurous. Training Chain-of-Thought via Latent-Variable Inference, 2023. 15

2023