{"id":"9cac3ef7-02ac-4969-abd8-36df00bab337","arxiv_id":"2412.08842","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Kajal uses GPT-3.5 with prompt engineering, few-shot examples, and parser feedback to infer grammars for DSL code snippets, reaching 60% self-parse accuracy on a 20-snippet dataset.","lead":"This paper presents Kajal, a tool that asks an LLM to generate a grammar for a code snippet and then refines the grammar using parser error messages, reporting 60% success with few-shot examples and 45% without. The result is a limited demonstration: success means the grammar parses the exact snippet it was built from, not that it works on new code from the same DSL.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PAP only tests that the generated grammar parses the exact snippet used to generate it; without held-out snippets per DSL, 60% does not demonstrate DSL grammar extraction.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the PAP metric and feedback loop validate a grammar only against the snippet that generated it, not against the DSL as a language. The paper's dataset of 20 snippets contains no held-out parse tests per DSL, and the iterative feedback uses the same snippet's parser errors to refine the grammar. This makes the evaluation circular with respect to the claim of grammar extraction. The paper does not compare against classical grammar induction baselines or use a reference grammar to generate test sets, further weakening the evidence. Given that the central claim depends on generalization to unseen code, and the reported metrics measure only self-parse fit, the reader's REJECT verdict is appropriate. No additional concerns are needed; the identified issue is sufficient to reject the paper's central claim as currently supported.","tokens_in":10180,"tokens_out":2995,"duration_ms":32981,"concrete_test":"For each DSL in the evaluation set, create or obtain at least three additional valid snippets from the same DSL (using the paper's definition or a known reference grammar). Run Kajal on the original snippet to infer a grammar, then parse the held-out snippets with that grammar and compute a held-out PAP. If this held-out PAP is close to 60%, the concern is resolved; if it is near 0%, the central claim is not supported by the current evaluation. This test should use the released artifacts to identify DSL groupings and generate held-out snippets independently of the LLM used in the paper.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Kajal extracts grammars of DSLs from code snippets. The evaluation, however, measures PAP as the percentage of snippets parsed by the grammar that was generated from each snippet itself. Section 3 describes the feedback loop: parser errors from the same input snippet are fed back to the LLM for up to ten iterations until that snippet parses. Thus a 'correct inference' means the grammar fits the single example used to produce it. The 20-snippet evaluation set (Section 4.1) provides no held-out snippets from the same DSLs, so there is no evidence that the inferred grammars accept other valid programs in those DSLs. Self-parse success could be achieved by a grammar that literally accepts the one snippet, which would not be a grammar of the DSL. The paper's own limitation list (Section 3) mentions semantic accuracy and feedback quality but not this fundamental evaluation gap. Consequently, the reported 60% with few-shot learning supports only 'the tool can fit a grammar to a snippet,' not 'the tool extracts the DSL grammar.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"Kajal is an end-to-end tool that prompts GPT-3.5 to generate a Lark grammar from a DSL code snippet, using three similar snippet-grammar pairs as few-shot examples and an iterative feedback loop that feeds parser errors from the input snippet back to the LLM for up to ten iterations. The paper evaluates the tool on 20 synthetic code snippets and reports that with few-shot learning 12 of 20 snippets yield grammars that parse those snippets (PAP = 60%), versus 9 of 20 (45%) without few-shot learning. The authors interpret these results as showing that Kajal can automatically extract DSL grammars and that few-shot learning significantly improves accuracy.","tokens_in":10333,"tokens_out":8092,"duration_ms":81826,"significance":"The goal of automating DSL grammar extraction is relevant, and the paper provides a complete, reproducible pipeline with released artifacts, which is a strength. The use of a fixed-cutoff model and the manual error analysis are also positive. However, the evaluation metric is circular: correctness is defined as parsing the exact snippet used to generate the grammar, and the feedback loop is driven by errors from that same snippet. Consequently, the reported accuracy measures per-snippet fitting rather than grammar generalization. If the authors were to re-evaluate with held-out snippets per DSL and a no-feedback control, the approach could be a useful contribution; in its current form, the central claim is not supported.","major_comments":[{"comment":"PAP (Section 4.3) counts a grammar as correct if it parses the code snippet that was used to generate it. Because the feedback loop in Section 3 feeds parser errors from that same snippet back to the LLM until parsing succeeds (or ten iterations elapse), a high PAP is the expected outcome of fitting the grammar to one example. The paper does not test the inferred grammars on any held-out snippets from the same DSLs, so the 60% figure does not provide evidence of DSL grammar extraction. This is the central evaluation gap.","section":"Section 4.3 and Section 3"},{"comment":"The evaluation dataset is described as 20 code snippets, but the paper does not state how many distinct DSLs these snippets come from or whether any DSL contributes more than one snippet. Without multiple snippets per DSL, there is no way to assess generalization, and the few-shot set (three snippets) is used only for prompt construction, not for validation. The authors should build a dataset with several snippets per DSL and report held-out accuracy.","section":"Section 4.1"},{"comment":"The ISRGC-n results are interpreted as showing that the iterative feedback loop is 'crucial,' but the paper does not include a control condition without feedback. Since the same snippet is used for both feedback and the success criterion, the improvement over iterations could be due to the LLM overfitting to the specific parser errors. A comparison against a single-shot (no-feedback) version is needed to support the claim.","section":"Section 4.4"}],"minor_comments":[{"comment":"The table header 'GIV' is inconsistent with the abbreviation 'GVI' used in Section 4.3 and Figure 5.","section":"Table 1"},{"comment":"The caption and axis label contain a typo, 'Iteartion Success Rate'; it should be 'Iteration Success Rate.'","section":"Figure 6"},{"comment":"The Similar Grammar Extractor is said to 'vectorize' snippets and use cosine similarity, but the vectorization method is not specified; this is needed for reproducibility.","section":"Section 3"},{"comment":"The paper uses 'few-shot learning' and 'few-shot prompting' without distinguishing them; as these are different techniques, the terminology should be clarified.","section":"Sections 2 and 3"},{"comment":"The evaluation does not discuss threats to validity such as prompt sensitivity, LLM nondeterminism, or the choice of parser; a short threats-to-validity paragraph would strengthen the paper.","section":"Section 4"},{"comment":"The code listings appear to have irregular spacing in the rendered PDF (e.g., 'im port'); please ensure the listings are typeset correctly.","section":"Listings 1 and 2"}],"recommendation":"reject","confidential_remarks":"The evaluation flaw is fundamental: the primary metric is self-parse success on the training snippet, so the paper's main quantitative claim is not established. I recommend rejection, but I would encourage the authors to revise the evaluation with held-out snippets per DSL and a no-feedback control; with that evidence the approach might be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Kajal is a clearly written, plausible LLM-based pipeline for inferring DSL grammars from code snippets, and the iteration-with-parser-feedback loop is a reasonable design. But the evaluation only shows the tool can fit a grammar to the exact snippet it was given; it never shows those grammars generalize to the DSL. So the headline 60% PAP is not evidence for the paper's central claim.\n\nWhat's genuinely new here is the specific combination: few-shot prompting with similar grammar examples, plus feeding Lark parser errors back to GPT-3.5 for up to ten rounds. The running example in Section 2 is easy to follow, and the pipeline is fully described. They also release artifacts, which I appreciate.\n\nThe load-bearing problem is the metric. PAP counts a grammar as correct iff it parses the input snippet, and the feedback loop uses errors from that same snippet to patch the grammar. So a successful parse is the expected outcome of fitting, not independent validation. The dataset is 20 synthetic snippets with no held-out snippets from the same DSLs, so we never see whether a \"correct\" grammar accepts other valid programs. Without that, 45% and 60% tell us about prompt engineering, not grammar induction. The paper even acknowledges related limitations (semantic accuracy, feedback quality) but not this one. The small sample, lack of error bars, and absence of any baseline comparison (e.g., Gramin or a simple grammar learner) make the quantitative claims hard to interpret. Also minor: the dataset was generated by ChatGPT-4o, and they don't verify that GPT-3.5 hasn't been exposed to similar DSLs despite the cut-off argument.\n\nWho is this for? Someone working on LLM-based parser construction might find the pipeline a useful starting point, and the failure analysis of Lark errors is interesting. But as it stands, the evidence doesn't support the claimed capability. I'd encourage a major revision with a proper held-out evaluation: for each DSL, generate multiple snippets; infer grammar from one; test on the others; also compare against a non-LLM baseline. That would turn a plausible demonstration into a real result.\n\nWorth sending to review rather than desk rejecting, because the idea is real and the flaw is fixable. I would not cite it in its current form.","headline":"A well-written LLM+parser-feedback pipeline for DSL grammar inference, but the evaluation measures only self-parse success, so the headline accuracy claims do not demonstrate grammar extraction.","tokens_in":10842,"tokens_out":2083,"would_cite":false,"duration_ms":22054,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Kajal claims that an LLM, given one DSL snippet plus three similar snippet-grammar examples and parser-error feedback, infers a grammar that parses the snippet correctly 60% of the time, versus 45% without the examples.","keywords":["grammar extraction","domain-specific languages","large language models","few-shot learning","prompt engineering","iterative refinement","Lark parser"],"falsifier":"Write a second program in the same DSL as one of the 12 snippets Kajal got right, do not put it in the prompt, and check whether the inferred grammar parses it; if many such second programs fail, the metric is measuring single-snippet fitting rather than DSL grammar learning.","tokens_in":9929,"feed_emoji":"🤖","tokens_out":8185,"duration_ms":78556,"temperature":0.7,"pith_summary":"Kajal tries to establish that grammar extraction for domain-specific languages need not be a manual, expert-heavy task: an LLM used through its API can write a parser grammar from a single code snippet, with a few similar examples and parser-error feedback. The headline numbers are 60% correctly parsed snippets with few-shot learning and 45% without, on 20 DSL snippets the model did not see. If the claim holds, anyone with a small or undocumented DSL could quickly obtain a first parseable grammar without labeled data, local training, or deep grammar expertise. The practical value depends on whether a grammar that parses one snippet also captures the DSL's general rules.","feed_headline":"LLM extracts DSL grammars from code at 60% accuracy","feed_subtitle":"Kajal feeds parser errors back to the model and reaches 60 percent on novel DSLs; no labeled training data needed.","key_machinery":"The machine is an iterative feedback loop around the LLM. Each candidate grammar is tested with the Lark parsing library; any parser error is returned as the user message in the next prompt, and the model revises its grammar until it parses the snippet or ten iterations are exhausted. Few-shot support supplies three similar code-snippet and grammar pairs, retrieved by cosine similarity over padded, vectorized snippets with a 0.5 threshold, to give the model a template for the expected grammar format. The prompt is structured with system instructions, user code, and assistant examples, and the model's output is wrapped in <GRAMMAR> tags so the parser component can extract it with regular expressions.","core_discovery":"The paper's central claim is that grammar induction for domain-specific languages can be moved from hand-written rules or bespoke algorithms to a prompt-engineering pipeline around a general-purpose LLM. Kajal selects the three most similar known snippets by cosine similarity with a threshold of 0.5, builds a prompt in the model's system and user fields, asks the model to emit a LARK grammar inside <GRAMMAR> tags, and then uses the Lark parser's error messages as feedback for up to ten refinement rounds. On the paper's 20-snippet evaluation set, with few-shot examples the tool produced grammars that parse their snippet in 12 of 20 cases (60%), with the other 8 failing; without few-shot examples, 9 of 20 (45%) parsed. The author concludes that few-shot learning materially improves grammar extraction and that the iterative feedback loop is essential, especially without few-shot.","pith_inferences":["Editorial: the paper's PAP metric checks only whether the inferred grammar parses the exact snippet it was built from, so the reported 60% does not yet demonstrate that the grammars generalize to other programs in the same DSL; a held-out-snippet test would settle that.","Editorial: because few-shot examples are chosen by cosine similarity with a single 0.5 threshold, the measured gain may be sensitive to that threshold; sweeping the threshold and the number of examples would show how much of the 15-point jump comes from the examples themselves versus the retrieval rule.","Editorial: the ten-round error-feedback loop runs at API level and produces parser errors as a free supervision signal; a natural testable extension is to collect snippet, error, and corrected-grammar triples and fine-tune a smaller open-source model, removing the dependence on a commercial API."],"forward_implications":["A developer can obtain a first parseable grammar for an undocumented DSL from a single example snippet, with no hand-written rules and no labeled training data.","Few-shot learning is the largest single lever in the pipeline: on the same 20 items it lifts both validity and correct-parsing rates from 45% to 60%.","The Lark parser provides free feedback signals, so the pipeline improves automatically without human annotation or a manually curated rule base.","The feedback loop's gains concentrate early, with most correct grammars produced in the first two iterations when few-shot learning is used, so a small iteration budget is enough.","The output grammars are standard LARK grammars, so they can feed directly into syntax highlighting, transpiling, or static-analysis tooling."],"supporting_citations":[{"why":"supplies the GPT-3.5 series model whose fixed knowledge cut-off is what makes the evaluation snippets plausibly unseen.","marker":"[19]"},{"why":"shows that few-shot prompting boosts inference accuracy, the core technique behind Kajal's prompt construction.","marker":"[14]"},{"why":"is cited as evidence that few-shot prompting improves inference for reasoning tasks, motivating Kajal's reliance on few-shot examples.","marker":"[20]"},{"why":"is the interactive grammar-extraction baseline that Kajal contrasts with, which needs user interaction and a knowledge base instead of an LLM.","marker":"[5]"},{"why":"is the Gramin incremental grammar-inference system, the main specialized prior work Kajal compares its feedback approach against.","marker":"[15]"},{"why":"is the semi-automatic grammar-recovery method from existing compiler artifacts, the classical approach Kajal claims to replace with direct LLM inference.","marker":"[10]"}],"fun_headline_variants":["LLM tool turns DSL code snippets into grammars, 60% accurate","Kajal automates grammar inference for DSLs via LLM, 60% success","Few-shot prompts and parser feedback boost LLM grammar induction to 60%","From code examples to LARK grammars: LLM pipeline hits 60%","LLM with iterative refinement extracts DSL grammars at 60% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a grammar that successfully parses the single snippet used to create it is the correct grammar of the underlying DSL; if the grammar only fits that one snippet, the 60% result does not show grammar extraction.","fun_headline_variants_meta":{"raw":{"variants":["LLM tool turns DSL code snippets into grammars, 60% accurate","Kajal automates grammar inference for DSLs via LLM, 60% success","Few-shot prompts and parser feedback boost LLM grammar induction to 60%","From code examples to LARK grammars: LLM pipeline hits 60%","LLM with iterative refinement extracts DSL grammars at 60% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000815,"raw_usage":{"total_tokens":3540,"prompt_tokens":880,"completion_tokens":2660,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2556}},"tokens_in":496,"tokens_out":2660,"duration_ms":18764,"temperature":1.0,"reasoning_tokens":2556,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:29:59.632389+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Write a second program in the same DSL as one of the 12 snippets Kajal got right, do not put it in the prompt, and check whether the inferred grammar parses it; if many such second programs fail, the metric is measuring single-snippet fitting rather than DSL grammar learning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is cited as evidence that few-shot prompting improves inference for reasoning tasks, motivating Kajal's reliance on few-shot examples."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the interactive grammar-extraction baseline that Kajal contrasts with, which needs user interaction and a knowledge base instead of an LLM."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"is the Gramin incremental grammar-inference system, the main specialized prior work Kajal compares its feedback approach against."}],"review_version":1}