Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Kajal: Extracting Grammar of a Source Code Using Large Language Models

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Kajal claims that an LLM, given one DSL snippet plus three similar snippet-grammar examples and parser-error feedback, infers a grammar that parses the snippet correctly 60% of the time, versus 45% without the examples.

desk verdict A well-written LLM+parser-feedback pipeline for DSL grammar inference, but the evaluation measures only self-parse success, so the headline accuracy claims do not demonstrate grammar extraction. read the letter →

arxiv 2412.08842 v1 pith:WONU3LJ2 submitted 2024-12-12 cs.SE cs.AI

Mohammad Jalili Torkamani This is my paper · ORCID
classification cs.SEcs.AI
keywords grammarextractiondomain-specificlanguageslargelanguagemodelsfew-shotlearningpromptengineeringiterativerefinementLarkparser
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Kajal tries to establish that grammar extraction for domain-specific languages need not be a manual, expert-heavy task: an LLM used through its API can write a parser grammar from a single code snippet, with a few similar examples and parser-error feedback. The headline numbers are 60% correctly parsed snippets with few-shot learning and 45% without, on 20 DSL snippets the model did not see. If the claim holds, anyone with a small or undocumented DSL could quickly obtain a first parseable grammar without labeled data, local training, or deep grammar expertise. The practical value depends on whether a grammar that parses one snippet also captures the DSL's general rules.

What carries the argument

The machine is an iterative feedback loop around the LLM. Each candidate grammar is tested with the Lark parsing library; any parser error is returned as the user message in the next prompt, and the model revises its grammar until it parses the snippet or ten iterations are exhausted. Few-shot support supplies three similar code-snippet and grammar pairs, retrieved by cosine similarity over padded, vectorized snippets with a 0.5 threshold, to give the model a template for the expected grammar format. The prompt is structured with system instructions, user code, and assistant examples, and the model's output is wrapped in <GRAMMAR> tags so the parser component can extract it with regular expressions.

What would settle it

Write a second program in the same DSL as one of the 12 snippets Kajal got right, do not put it in the prompt, and check whether the inferred grammar parses it; if many such second programs fail, the metric is measuring single-snippet fitting rather than DSL grammar learning.

Watch

Extended reading notes

Core claim

The paper's central claim is that grammar induction for domain-specific languages can be moved from hand-written rules or bespoke algorithms to a prompt-engineering pipeline around a general-purpose LLM. Kajal selects the three most similar known snippets by cosine similarity with a threshold of 0.5, builds a prompt in the model's system and user fields, asks the model to emit a LARK grammar inside <GRAMMAR> tags, and then uses the Lark parser's error messages as feedback for up to ten refinement rounds. On the paper's 20-snippet evaluation set, with few-shot examples the tool produced grammars that parse their snippet in 12 of 20 cases (60%), with the other 8 failing; without few-shot examples, 9 of 20 (45%) parsed. The author concludes that few-shot learning materially improves grammar extraction and that the iterative feedback loop is essential, especially without few-shot.

Load-bearing premise

The load-bearing premise is that a grammar that successfully parses the single snippet used to create it is the correct grammar of the underlying DSL; if the grammar only fits that one snippet, the 60% result does not show grammar extraction.

Editorial extensions

If this is right

  • A developer can obtain a first parseable grammar for an undocumented DSL from a single example snippet, with no hand-written rules and no labeled training data.
  • Few-shot learning is the largest single lever in the pipeline: on the same 20 items it lifts both validity and correct-parsing rates from 45% to 60%.
  • The Lark parser provides free feedback signals, so the pipeline improves automatically without human annotation or a manually curated rule base.
  • The feedback loop's gains concentrate early, with most correct grammars produced in the first two iterations when few-shot learning is used, so a small iteration budget is enough.
  • The output grammars are standard LARK grammars, so they can feed directly into syntax highlighting, transpiling, or static-analysis tooling.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the paper's PAP metric checks only whether the inferred grammar parses the exact snippet it was built from, so the reported 60% does not yet demonstrate that the grammars generalize to other programs in the same DSL; a held-out-snippet test would settle that.
  • Editorial: because few-shot examples are chosen by cosine similarity with a single 0.5 threshold, the measured gain may be sensitive to that threshold; sweeping the threshold and the number of examples would show how much of the 15-point jump comes from the examples themselves versus the retrieval rule.
  • Editorial: the ten-round error-feedback loop runs at API level and produces parser errors as a free supervision signal; a natural testable extension is to collect snippet, error, and corrected-grammar triples and fine-tune a smaller open-source model, removing the dependence on a commercial API.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. Kajal is an end-to-end tool that prompts GPT-3.5 to generate a Lark grammar from a DSL code snippet, using three similar snippet-grammar pairs as few-shot examples and an iterative feedback loop that feeds parser errors from the input snippet back to the LLM for up to ten iterations. The paper evaluates the tool on 20 synthetic code snippets and reports that with few-shot learning 12 of 20 snippets yield grammars that parse those snippets (PAP = 60%), versus 9 of 20 (45%) without few-shot learning. The authors interpret these results as showing that Kajal can automatically extract DSL grammars and that few-shot learning significantly improves accuracy.

Significance. The goal of automating DSL grammar extraction is relevant, and the paper provides a complete, reproducible pipeline with released artifacts, which is a strength. The use of a fixed-cutoff model and the manual error analysis are also positive. However, the evaluation metric is circular: correctness is defined as parsing the exact snippet used to generate the grammar, and the feedback loop is driven by errors from that same snippet. Consequently, the reported accuracy measures per-snippet fitting rather than grammar generalization. If the authors were to re-evaluate with held-out snippets per DSL and a no-feedback control, the approach could be a useful contribution; in its current form, the central claim is not supported.

major comments (3)
  1. [Section 4.3 and Section 3] PAP (Section 4.3) counts a grammar as correct if it parses the code snippet that was used to generate it. Because the feedback loop in Section 3 feeds parser errors from that same snippet back to the LLM until parsing succeeds (or ten iterations elapse), a high PAP is the expected outcome of fitting the grammar to one example. The paper does not test the inferred grammars on any held-out snippets from the same DSLs, so the 60% figure does not provide evidence of DSL grammar extraction. This is the central evaluation gap.
  2. [Section 4.1] The evaluation dataset is described as 20 code snippets, but the paper does not state how many distinct DSLs these snippets come from or whether any DSL contributes more than one snippet. Without multiple snippets per DSL, there is no way to assess generalization, and the few-shot set (three snippets) is used only for prompt construction, not for validation. The authors should build a dataset with several snippets per DSL and report held-out accuracy.
  3. [Section 4.4] The ISRGC-n results are interpreted as showing that the iterative feedback loop is 'crucial,' but the paper does not include a control condition without feedback. Since the same snippet is used for both feedback and the success criterion, the improvement over iterations could be due to the LLM overfitting to the specific parser errors. A comparison against a single-shot (no-feedback) version is needed to support the claim.
minor comments (6)
  1. [Table 1] The table header 'GIV' is inconsistent with the abbreviation 'GVI' used in Section 4.3 and Figure 5.
  2. [Figure 6] The caption and axis label contain a typo, 'Iteartion Success Rate'; it should be 'Iteration Success Rate.'
  3. [Section 3] The Similar Grammar Extractor is said to 'vectorize' snippets and use cosine similarity, but the vectorization method is not specified; this is needed for reproducibility.
  4. [Sections 2 and 3] The paper uses 'few-shot learning' and 'few-shot prompting' without distinguishing them; as these are different techniques, the terminology should be clarified.
  5. [Section 4] The evaluation does not discuss threats to validity such as prompt sensitivity, LLM nondeterminism, or the choice of parser; a short threats-to-validity paragraph would strengthen the paper.
  6. [Listings 1 and 2] The code listings appear to have irregular spacing in the rendered PDF (e.g., 'im port'); please ensure the listings are typeset correctly.

Circularity Check

2 steps flagged · score 8.0 of 10

PAP measures only whether the inferred grammar parses the exact snippet it was generated from; no held-out snippets per DSL are used, so the reported 45%/60% accuracy is a per-example fit rate, not grammar extraction.

  1. self definitional [Section 2 (Example) and Section 3 (Technical Details), feedback loop]
    "If parsing succeeds, the record is marked as a correct inferred grammar, and the process moves to the next dataset record. ... This feedback loop continues for up to ten iterations, providing the error message for each unsuccessful inference until a correct grammar (which can parse the given code snippet from the dataset) is generated or the attempt is marked as failed."

    The grammar is generated from a prompt built from the input code snippet, and the same snippet is the sole success criterion. 'Correct' therefore denotes 'accepts the single example used in the prompt.' The feedback loop is driven by parser errors on that same snippet, so the loop is fitting the grammar to the example. A grammar that merely matches or enumerates the one snippet would be counted as correct. This is not a test that the grammar generates the DSL's language, since no other sentence of the DSL is checked.

  2. fitted input called prediction [Section 4.3 Evaluation Metrics; Section 4.4 Results and Analysis]
    "PAP = Number of Correctly Parsed Code Snippets / Total Number of Code Snippets × 100 ... This metric reflects the percentage of code snippets for which Kajal inferred the correct grammar. It indicates the accuracy and efficiency of our approach in generating grammars that can parse the given code snippet."

    The numerator is exactly the condition used by the iterative generator: the grammar must parse the snippet that was placed in the prompt. Table 1's 'Correct Inferences' are therefore self-parses. The 20 evaluation records are the snippets used for inference, and Section 4.1 describes only an evaluation set and a few-shot learning set, with no held-out snippets from the same DSLs. Calling PAP 'accuracy' for grammar extraction converts a per-example fit success rate into a claim about DSL grammars, which is a fitted input presented as performance.

full rationale

The central claim of the paper is that Kajal automatically extracts DSL grammars from code snippets, with 60% correctness under few-shot learning and 45% without it. However, the paper's own definition of correctness is that the inferred grammar parses the input snippet used to generate it, and the feedback loop iterates on parser errors from that same snippet until it parses or the attempt fails. Thus the reported metric is a measure of how often the tool can fit a grammar to the single example it was given, not whether the grammar generalizes to other valid programs in the DSL. The paper describes no held-out snippets per DSL, so no external benchmark checks whether the inferred grammars accept more than the training example. This is a definitional circularity in the primary evaluation: the success condition is the target used to drive generation. There are no load-bearing self-citations or imported uniqueness theorems here; the circularity lies in the evaluation design. The Section 3 limitation notes mention semantic accuracy and feedback quality, but they do not address the absence of held-out validation, so they do not mitigate the issue. A score of 8 rather than 10 reflects that there is still nontrivial engineering in producing a syntactically valid Lark grammar, but the headline accuracy number reduces by construction to self-parse success.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The tool itself is not an invented entity in the sense of a new physical or theoretical object. The main burden is the evaluation axiom that equates self-parse with correctness, plus hand-set hyperparameters. No new theoretical entities are introduced.

free parameters (3)
  • Number of few-shot examples = 3
    Three similar snippets are selected for every prompt (Section 3). No ablation beyond zero vs. three.
  • Cosine similarity threshold = 0.5
    Minimum similarity for choosing few-shot examples, set in Section 3 without empirical tuning or justification.
  • Maximum feedback iterations = 10
    The feedback loop is capped at ten iterations (Section 3). No analysis of the effect of this cap.
assumptions (5)
  • domain assumption The evaluation snippets, generated by ChatGPT-4o after GPT-3.5's cut-off, were not seen by GPT-3.5 during training.
    Used in Section 4.1 to argue the evaluation is unbiased. Assumes no data leakage through shared training corpora or indirect exposure.
  • domain assumption Cosine similarity over padded vectorized code snippets identifies useful examples for few-shot learning.
    Invoked in Section 3 to select three similar snippets; no evidence that this similarity correlates with grammar similarity.
  • ad hoc to paper A grammar that parses the single input snippet is a correct grammar for the DSL.
    This is the central evaluation assumption behind PAP in Section 4.3 and Section 4.4; it equates self-parse success with grammar correctness.
  • domain assumption Lark's grammar syntax and LALR parsing are suitable for representing and validating inferred grammars.
    Used in the Parser component in Section 3; assumes any useful DSL grammar is expressible in Lark's grammar language.
  • domain assumption The LLM's output inside <GRAMMAR> tags can be reliably extracted with regular expressions.
    Section 3 says grammar is extracted using regex; assumes the model consistently follows the output format.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Kajal: Extracting Grammar of a Source Code Using Large Language Models." pith.science (2026). https://pith.science/paper/WONU3LJ2

@misc{pith2026241208842,
  author       = {Pith},
  title        = {Pith review of: Kajal: Extracting Grammar of a Source Code Using Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WONU3LJ2}},
  note         = {Machine review of arXiv:2412.08842}
}
read the original abstract

Understanding and extracting the grammar of a domain-specific language (DSL) is crucial for various software engineering tasks; however, manually creating these grammars is time-intensive and error-prone. This paper presents Kajal, a novel approach that automatically infers grammar from DSL code snippets by leveraging Large Language Models (LLMs) through prompt engineering and few-shot learning. Kajal dynamically constructs input prompts, using contextual information to guide the LLM in generating the corresponding grammars, which are iteratively refined through a feedback-driven approach. Our experiments show that Kajal achieves 60% accuracy with few-shot learning and 45% without it, demonstrating the significant impact of few-shot learning on the tool's effectiveness. This approach offers a promising solution for automating DSL grammar extraction, and future work will explore using smaller, open-source LLMs and testing on larger datasets to further validate Kajal's performance.

Figures

Figures reproduced from arXiv: 2412.08842 by the authors.

Figure 1
Figure 1. Kajal Workflow. s t a r t : s t a t e m e n t + s t a t e m e n t : " Kadd " ex p r " , " ex p r −> add | " Ksub " ex p r " , " ex p r −> s u b t r a c t | " Kmul " ex p r " , " ex p r −> m u l t i p l y | " Kdiv " ex p r " , " ex p r −> d i v i d e | "Kmod " ex p r " , " ex p r −> mod | " Kpow " ex p r " , " ex p r −> power | " K s q r t " ex p r −> s q r t | " Klog " ex p r −> l o g | " K fa c " ex p r −> f a c t … view at source ↗
Figure 2
Figure 2. System Field. user-assistant pairs, enabling the model to learn from limited examples without requiring additional training data. After constructing the LLM fields (system, user, and as￾sistant), the input is passed to the LLM component. This component uses the GPT-3 model (with default configura￾tions) to perform inference. The GPT-3 model is selected due to its fixed knowledge cut-off date, ensuring that it does n… view at source ↗
Figure 3
Figure 3. illustrates an example of inference made by Kajal. The parser also tests the grammar by attempting to parse the input code snippet using the Lark library with an lalr parser. Then, each inference result is evaluated by the Evaluator component based on predefined metrics (detailed in Section 4). All results are saved as a single comprehensive JSON file for further analysis. One key feature of Kajal is its feedback-dr… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Kajal Experiments’ Results [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Iteartion Success Rate. 4.4.2 ISRGC / Without Few-Shot Learning. : Without few-shot learning, the model shows a slightly different be￾havior. The first iteration produces 5 correct inferences, but there is a sharp drop to 1 correct inference in the second iter￾ation, a…
Figure 5
Figure 5. Figure 5: Kajal GIV and PAP Metrics [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Struct-Bench: A Benchmark for Differentially Private Structured Text Generation

    cs.CL 2025-09 conditional novelty 7.0 of 10

    Struct-Bench is a Context-Free Grammar based evaluation platform showing that current differentially private synthetic text generators often fail to preserve structure and semantic diversity.

Reference graph

Works this paper leans on

22 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al

  2. [2]

    Martin Fowler. 2010. Domain-specific languages. Pearson Education

    2010

  3. [3]

    Dick Grune, Kees Van Reeuwijk, Henri E Bal, Ceriel JH Jacobs, and Koen Langendoen. 2012. Modern compiler design. Springer Science & Business Media

    work page 2012

  4. [4]

    Rahul Jain, Sanjeev Kumar Aggarwal, Pankaj Jalote, and Shiladitya Biswas. 2004. An interactive method for extracting grammar from programs. Software: Practice and Experience 34, 5 (2004), 433–447

    work page 2004

  5. [5]

    Rahul Jain, Sanjeev Kumar Aggarwal, Pankaj Jalote, and Shi- laditya Biswas. 2004. An interactive method for extract- ing grammar from programs. Software: Practice and Expe- rience 34, 5 (2004), 433–447. https://doi.org/10.1002/spe.568 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/spe.568

    work page doi:10.1002/spe.568 2004

  6. [6]

    Kieffer and En-Hui Yang

    J.C. Kieffer and En-Hui Yang. 2000. Grammar-based codes: a new class of universal lossless source codes. IEEE Transactions on Information Theory 46, 3 (2000), 737–754. https://doi.org/10.1109/18.841160

    doi:10.1109/18.841160 2000

  7. [7]

    Ralf Lämmel and Chris Verhoef. 2001. Semi-automatic grammar re- covery. Software: Practice and Experience 31, 15 (2001), 1395–1438

    work page 2001

  8. [8]

    Triet H. M. Le, Hao Chen, and Muhammad Ali Babar. 2020. Deep Learning for Source Code Modeling and Generation: Models, Appli- cations, and Challenges. ACM Comput. Surv. 53, 3, Article 62 (June 2020), 38 pages. https://doi.org/10.1145/3383458

    work page doi:10.1145/3383458 2020

Show all 22 references
  1. [9]

    Wolfgang Lohmann. 2009. On language processors and software main- tenance. Ph. D. Dissertation. Citeseer. 1https://github.com/mohammadJaliliTorkamani/Kajal

    2009

  2. [10]

    Lämmel and C

    R. Lämmel and C. Verhoef. 2001. Semi-automatic gram- mar recovery. Software: Practice and Experience 31, 15 (2001), 1395–1438. https://doi.org/10.1002/spe.423 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1002/spe.423

    2001 doi

  3. [11]

    Ronald Mak. 2011. Writing compilers and interpreters: a software engi- neering approach. John Wiley & Sons

    2011

  4. [12]

    David Méndez-Acuña, José A Galindo, Benoit Combemale, Arnaud Blouin, Benoit Baudry, and Gurvan Le Guernic. 2016. Reverse- engineering reusable language modules from legacy domain-specific languages. In Software Reuse: Bridging with Social-A wareness: 15th International Confere...

    2016

  5. [13]

    Marjan Mernik, Jan Heering, and Anthony M. Sloane. 2005. When and how to develop domain-specific languages. ACM Comput. Surv. 37, 4 (Dec. 2005), 316–344. https://doi.org/10.1145/1118890.1118892

    2005

  6. [14]

    Elliot Meyerson, Mark J Nelson, Herbie Bradley, Adam Gaier, Arash Moradi, Amy K Hoover, and Joel Lehman. 2023. Language model crossover: Variation through few-shot prompting. arXiv preprint arXiv:2302.12170 (2023)

    2023

  7. [15]

    Diptikalyan Saha and Vishal Narula. 2011. Gramin: a system for incre- mental learning of programming language grammars. In International Symposium on Electronic Commerce . https://api.semanticscholar.org/ CorpusID:1743193

    2011

  8. [16]

    Masahiro Suzuki, Hiroki Sakaji, Masanori Hirano, and Kiyoshi Izumi

  9. [17]

    Mihai VIERU, Vlad POLISCIUC, Daniela GLIGA, Ecaterina GREBEN- NICOVA, and Anastasia ZAGORODNIUC. 2024. Domain-specific language for analyzing medical results. (2024)

    2024

  10. [18]

    William M Waite and Gerhard Goos. 2012. Compiler construction . Springer Science & Business Media

    2012

  11. [19]

    Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al . 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420 (2023)

    2023 arXiv

  12. [20]

    Xi Ye and Greg Durrett. 2022. The unreliability of explanations in few- shot prompting for textual reasoning. Advances in neural information processing systems 35 (2022), 30378–30392

    2022

  13. [2023]

    Information Processing & Management 60, 2 (2023), 103194

    Constructing and analyzing domain-specific language model for financial text mining. Information Processing & Management 60, 2 (2023), 103194

    2023

  14. [2024]

    ACM Trans- actions on Intelligent Systems and Technology 15, 3 (2024), 1–45

    A survey on evaluation of large language models. ACM Trans- actions on Intelligent Systems and Technology 15, 3 (2024), 1–45

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.