Pith. sign in

REVIEW 4 major objections 4 minor 17 references

Inferring Attributed Grammars from Parser Implementations

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A parser's runtime behavior can be lifted into an attributed grammar, recovering both syntax and semantics from code alone.

desk verdict Novel dynamic tracing approach for semantic grammar recovery, but the 'comprehensive specification' claim outruns the happy-path evaluation. read the letter →

arxiv 2507.13117 v1 pith:HCE4HRLZ submitted 2025-07-17 cs.SE

classification cs.SE
keywords attributedgrammarinferencespecificationrecoverysemanticactionextractionrecursivedescentparserdynamicprogramanalysisparsetreemappingfuzzinginputgrammars
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Legacy systems that parse structured input often lack a written specification of what the input means or how it is processed. The paper claims that this missing semantic specification can be recovered automatically: starting from a syntactic grammar and a recursive descent parser implementation, it runs the parser on generated inputs, records the runtime parse tree, aligns that tree with the grammar's derivation tree, and lifts the parser's semantic statements into the grammar as attributes and semantic rules. The result is an attributed grammar, a single executable description of both syntax and input-to-output behavior. Preliminary results show that for four of seven small benchmarks the reconstructed grammar reproduces the original program's output on every tested input, with the remaining three failing on exception handling, uncovered functions, or misplacement of one semantic rule.

What carries the argument

The load-bearing object is the node-mapping table between the runtime parse tree and the grammar derivation tree, together with the terminal-yield range attached to each node. In the parse tree, input-character accesses, function calls, loops, and conditionals become nodes; in the derivation tree, terminals, nonterminals, repetitions, options, and choices become nodes. Mapping proceeds by matching terminal sequences first, then matching functions to nonterminals bottom-up, then matching control-flow nodes to grammar structures within each function's scope; unmapped semantics are pushed to nearby children. This table is what lets ordinary imperative code be transferred into formal semantic actions at the correct positions in the grammar.

What would settle it

Run both the original parser and the reconstructed attributed grammar on inputs that exercise exception handling or uncovered functions; the paper reports such inputs already exist for MathExpr (13 of 32 outputs disagree) and MicroJson (9 of 22 outputs disagree). Any such mismatch shows that the recovered grammar is not yet a complete behavioral specification for inputs beyond the covered execution paths.

Watch

Extended reading notes

Core claim

The central claim is that an attributed grammar can be inferred from a recursive descent parser by dynamic analysis, provided that a syntactic grammar for the same input language is already available. The key step is a type-and-yield mapping between two trees built from the same input: the parse tree produced by observing the running parser and the derivation tree produced by parsing that input with the grammar. When the mapping succeeds, code that is not part of syntax checks is grouped into semantic blocks and inserted at the corresponding grammar positions, with function calls, parameters, return statements, and assignments translated into attribute rules. The paper argues that the resulting attributed grammar faithfully reproduces the original program's input-processing behavior, and the evaluation reports exact output agreement on four of seven benchmarks.

Load-bearing premise

The method assumes that the shape of the running parser mirrors the shape of the grammar, so that every parser function matches a grammar rule, every loop matches a repetition, and every condition matches an option or choice covering the same input range; when parser and grammar diverge structurally, the mapping fails.

Editorial extensions

If this is right

  • For parsers that follow the assumed recursive descent style, specification recovery can cover semantics as well as syntax, turning the recovered attributed grammar into an executable specification of input processing.
  • Because the recovered attributed grammar is executable, it can be used to generate inputs with known expected outputs, providing a test oracle for the original parser and for future modifications.
  • The mined grammar can be read by humans as documentation of a parser's behavior, filling a gap left by missing or outdated specifications.
  • The method currently captures only happy-path semantics: exception and error-handling behavior is not recovered, and inputs must avoid runtime exceptions such as evaluating asin(x) outside the valid domain.
  • The mapping tolerates only minor structural differences between parser and grammar; recursion instead of iteration, grammar rule indirections, or major control-flow deviations are not handled.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: if the mapping were extended with grammar normalization, for example by converting iteration to recursion or eliminating left recursion, the same trace-alignment idea could cover many hand-written parsers that do not literally mirror the supplied grammar.
  • Editorial inference: the recovered semantic actions could support differential testing, where the original parser and the generated attributed grammar are both run on fuzzed inputs and any output disagreement flags a behavioral difference.
  • Editorial inference: combining the approach with symbolic or concolic execution to reach exception-raising inputs would address the paper's stated happy-path limitation and let error-handling semantics be captured.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a dynamic-analysis technique for enriching an existing context-free grammar of a recursive-descent parser with semantic actions, yielding an attributed grammar. Given a grammar and a parser program, the method generates syntactically valid inputs, executes the parser to obtain a runtime parse tree, derives a derivation tree from the grammar, maps nodes between the two trees under a type-and-yield-range isomorphism assumption, and transfers semantic code from the program into the grammar. The paper presents a running example (a number parser), a small evaluation of seven parser benchmarks from the literature, and reports output-agreement accuracies ranging from 59% to 100%. The author's central claim is that the recovered attributed grammar reproduces the program's input-processing behavior and constitutes a comprehensive specification.

Significance. If the approach worked as stated, it would be a useful contribution to specification mining: prior grammar-mining work mostly recovers syntax, while this paper targets the under-explored problem of recovering semantic actions. The proposed mapping between parse-tree node types and grammar constructs is concrete, and the running example demonstrates the intended mechanism. However, the current evidence supports only a narrow feasibility claim: the evaluation uses a small set of manually curated examples, two benchmarks reach only 59% accuracy, and the paper's own limitations section concedes that exceptional and error-handling behavior is not modeled. The significance is therefore preliminary rather than established, and the paper's stronger wording, such as "comprehensive specification recovery," is not justified by the presented results. I also note that no artifact or reproducibility package is mentioned, which limits independent verification.

major comments (4)
  1. [Abstract, §I, §III, §V] The central specification-recovery claim is contradicted by the paper's own scope statement. The abstract and Section I assert that the approach yields "comprehensive specification recovery" and "complete attributed grammars that capture both syntax and semantics," but Section V states that the method captures only happy-path semantics and does not track exceptional or error-handling behavior. Since a parser's behavior includes rejecting malformed inputs and raising exceptions, an attributed grammar that omits those behaviors is not behaviorally equivalent to the original program on all inputs. The accuracy measurement in Section IV only compares outputs on inputs that are valid by construction, so it cannot support the unqualified equivalence claim I → AG → O made in Section III. Please rephrase the contribution as recovery of happy-path semantic actions, or extend the method to exception paths and demonstrate that equivalence on invalid inputs as well.
  2. [§III-A, §IV, Table IV] The evaluation does not exercise the input-generation step described in the method. Section III-A says that grammar fuzzing combining the Efficient Grammar Fuzzer and Grammarinator's cooldown strategy is used to achieve grammar coverage, but Section IV states "we manually crafted inputs to achieve full statement coverage." Consequently, the reported accuracies are evidence for the mapping and transfer steps on hand-picked inputs only; the fuzzing pipeline is never evaluated. Moreover, MathExpr and MicroJson reach only 59% accuracy, and the text's explanation ("certain functions were not fully covered," "an incorrect placement of a semantic rule") does not quantify which grammar rules or program paths were covered or missed. Please either run and report the fuzzing-based end-to-end pipeline, or explicitly present the evaluation as a manually curated feasibility study with per-benchmark coverage data.
  3. [§III-D, §III-E, §IV] The accuracy metric is close to an internal consistency check. Semantic actions embedded into the attributed grammar are, by construction, verbatim statements of the program's own code, and the correctness test compares the outputs of the attributed-grammar interpreter with outputs of the same program. High agreement therefore shows that the extraction-and-placement pipeline preserved the original statements on the tested inputs, not that the inferred semantics are correct against an independent specification. For example, if the mapping misplaced a semantic action, the test would detect it on the exercised trace, but it would not detect a semantic action that is itself an incorrect implementation of the intended language semantics. The paper should state this distinction explicitly and add at least one validation that is not purely transcription fidelity, such as checking the recovered attributed grammar against a separately written reference specification for one of the benchmarks.
  4. [§III-D, §V] The structural-isomorphism assumption behind the node mapping is load-bearing and is acknowledged to be narrow. Section III-D assumes that input-character nodes correspond to terminals, function nodes to nonterminals, loops to repetitions, and conditions to options or choices, with corresponding nodes covering the same input ranges. Section V then concedes that the method "does not handle substantial deviations such as grammar rule indirections, function calls, or major control flow differences" and that some examples from prior work had to be adapted to fit the format. Given this concession, the general framing about "recursive descent parsers" in Section I overstates the applicability. The paper should characterize the class of parsers the method actually covers (e.g., one-function-per-grammar-rule, no exception handling inside parsing functions, no recursion-versus-repetition mismatches) and state this assumption prominently in the introduction and abstract.
minor comments (4)
  1. [§III-C, Listing 1] The line references in the text do not match the line numbering of Listing 1; for example, the root node Number is attributed to "line 1" while the function definition appears on line 4 of the listing, and the conditional in the text's "line 3" appears on line 6 of the listing. Please align these references or explain the numbering scheme.
  2. [Figure 2] The caption says that the subscript numbers represent the terminal yield, but the subscripts are intervals such as [1-3]; please clarify that these are input position ranges covered by the node, and state whether intervals are inclusive.
  3. [§III-D.2] The bottom-up function-to-nonterminal mapping algorithm is described informally; in particular, the termination condition and the behavior when no node with matching terminal yield exists are not defined. A short pseudocode listing or a precise invariant would make the algorithm reproducible.
  4. [References] Reference [16] contains the stray word "unread" in the bibliographic entry, and the affiliation line "Institut for System Software" appears to be a typo for "Institute." These should be corrected.

Circularity Check

1 steps flagged · score 4.0 of 10

Accuracy evaluation is a self-consistency check because semantic actions are copied from the program into the grammar, though the node-mapping step is non-trivial and reported failures keep the method from being fully circular.

  1. self definitional [Section III.E (Transfer of Semantic Actions) and Section IV (Preliminary Evaluation)]
    "'All code elements that appear between the blocks responsible for syntax analysis are considered semantic. These semantic elements are grouped into blocks and placed at the appropriate positions in the grammar according to the established mapping between parse tree and derivation tree nodes.' ... 'we run both the original program and an interpreter for the reconstructed attributed grammar on the same inputs and compare their serialized outputs. Accuracy is measured as the proportion of matching outputs, indicating how well the grammar replicates program behavior.'"

    The AG's semantic actions are the program's own statements moved into grammar positions; hence the AG's output behavior is defined by the program P. Running AG and P on the same inputs and comparing outputs compares a transcription to its source: whenever the mapping is correct, agreement is forced by construction. Table IV therefore measures mapping and placement fidelity, not whether an independently recovered specification captures the program. The reported failures in MathExpr, MicroJson, and SimpleExpr show the mapping is non-trivial, but the accuracy metric still cannot independently validate the abstract's 'comprehensive specification recovery' claim.

full rationale

The core technical contribution—mapping parse-tree nodes to derivation-tree nodes and placing semantic actions at grammar positions—is non-trivial and has independent content; the method fails on three of seven benchmarks, so the mapping is not vacuous. However, the paper's validation is internally circular: Section III.E defines the AG's semantics as the program's statements transferred into the grammar, and Section IV evaluates the AG by running it and the original program on the same inputs. Under a correct mapping, output agreement is guaranteed by construction, so a 100% row demonstrates transcription fidelity rather than independent specification recovery. This circularity is partial because the mapping step can and does fail. In addition, Section V explicitly concedes that the method 'currently captures only semantics along the happy path' and that 'semantics related to exceptional or error-handling behavior are not tracked'; this is a limitation of the recovered artifact, not itself a circular step, but it further weakens the 'comprehensive specification recovery' language in the Abstract and Section I. No load-bearing self-citation chain or fitted-input-as-prediction pattern is present; the reuse of SimpleExpr from prior work is disclosed and is not central to the argument. Score 4 reflects a real but partial circularity in the evaluation, not a fully forced derivation.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests on domain assumptions about parser structure and grammar availability rather than on fitted parameters or new entities. The main assumptions are that the target is a recursive descent parser with one function per rule, that a correct syntax grammar is already given, that runtime nodes can be matched to grammar nodes by type and yield, that high-coverage inputs can be generated, and that program statements can be transcribed into synthesized attributes. None of these are formally verified or evidenced beyond the seven small examples.

assumptions (5)
  • domain assumption The parser under analysis is a recursive descent parser with a one-to-one mapping from grammar rules to functions.
    Stated as a precondition in Section III and in Section V: 'The method is also limited to programs that strictly follow a recursive descent parser structure.'
  • domain assumption A syntactic grammar G for the program exists and is aligned with the parser's structure.
    Section III: 'Our approach assumes the existence of a syntactic grammar, which may have been previously mined.'
  • domain assumption Runtime parse tree nodes correspond to derivation tree nodes when types and terminal yields match.
    Section III-D: 'we assume that Input Character nodes in the parse tree correspond to Terminal nodes... function nodes correspond to non-terminal symbols...'
  • domain assumption Inputs generated by grammar fuzzing achieve sufficient coverage of grammar elements and positions.
    Section III-A states the input set 'must provide sufficient coverage of the grammar, including all terminal and non-terminal symbols, production rules, and positions within rules.' In the evaluation, however, inputs were manually curated, so this assumption is untested.
  • domain assumption The semantic effect of a parser can be expressed as synthesized attributes in an attributed grammar.
    The translation table in Section III-E maps assignments, returns, and function calls to attribute assignments, assuming this representation is complete.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Inferring Attributed Grammars from Parser Implementations." pith.science (2026). https://pith.science/paper/HCE4HRLZ

@misc{pith2026250713117,
  author       = {Pith},
  title        = {Pith review of: Inferring Attributed Grammars from Parser Implementations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HCE4HRLZ}},
  note         = {Machine review of arXiv:2507.13117}
}
read the original abstract

Software systems that process structured inputs often lack complete and up-to-date specifications, which specify the input syntax and the semantics of input processing. While grammar mining techniques have focused on recovering syntactic structures, the semantics of input processing remains largely unexplored. In this work, we introduce a novel approach for inferring attributed grammars from parser implementations. Given an input grammar, our technique dynamically analyzes the implementation of recursive descent parsers to reconstruct the semantic aspects of input handling, resulting in specifications in the form of attributed grammars. By observing program executions and mapping the program's runtime behavior to the grammar, we systematically extract and embed semantic actions into the grammar rules. This enables comprehensive specification recovery. We demonstrate the feasibility of our approach using an initial set of programs, showing that it can accurately reproduce program behavior through the generated attributed grammars.

Figures

Figures reproduced from arXiv: 2507.13117 by the authors.

Figure 2
Figure 2. The parse tree on the left side and the derivation tree on the right [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

17 extracted references · 12 canonical work pages

  1. [1]

    Little languages: little maintenance?

    A. V . Deursen and P. Klint, “Little languages: little maintenance?” Journal of Software Maintenance: Research and Practice , vol. 10, no. 2, p. 75–92, Mar. 1998. [Online]. Available: http://dx.doi.org/10. 1002/(SICI)1096-908X(199803/04)10:2⟨75::AID-SMR168⟩3.0.CO;2-5

  2. [2]

    Grammars for free: toward grammar inference for Ad Hoc parsers,

    M. Schr ¨oder and J. Cito, “Grammars for free: toward grammar inference for Ad Hoc parsers,” inProceedings of the ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results . Pittsburgh Pennsylvania: ACM, May 2022, pp. 41–45. [Online]. Available: https://dl.acm.org/doi/10.1145/3510455.3512787

  3. [3]

    Semantics of context-free languages,

    D. E. Knuth, “Semantics of context-free languages,” Mathematical Systems Theory , vol. 2, no. 2, pp. 127–145, Jun. 1968. [Online]. Available: http://link.springer.com/10.1007/BF01692511

  4. [4]

    A survey of grammatical inference in software engineering,

    A. Stevenson and J. R. Cordy, “A survey of grammatical inference in software engineering,” Science of Computer Programming , vol. 96, pp. 444–459, Dec. 2014. [Online]. Available: https://linkinghub.elsevier. com/retrieve/pii/S0167642314002469

  5. [5]

    Mining input grammars from dynamic control flow,

    R. Gopinath, B. Mathis, and A. Zeller, “Mining input grammars from dynamic control flow,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Virtual Event USA: ACM, Nov. 2020, pp. 172–183. [Online]. Available: https://dl.acm.org/doi/10.1145/3368089.3409679

  6. [6]

    Mining input grammars from dynamic taints,

    M. H ¨oschele and A. Zeller, “Mining input grammars from dynamic taints,” in Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering . Singapore Singapore: ACM, Aug. 2016, pp. 720–725. [Online]. Available: https://dl.acm.org/doi/10. 1145/2970276.2970321

  7. [7]

    Towards Attribute Grammar Mining by Symbolic Execution,

    M. Moser, J. Pichler, and A. Pointner, “Towards Attribute Grammar Mining by Symbolic Execution,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER). Honolulu, HI, USA: IEEE, Mar. 2022, pp. 822–826. [Online]. Available: https://ieeexplore.ieee.org/document/9825876/

  8. [8]

    Mining Attributed Input Grammars and their Applications in Fuzzing,

    A. Pointner, “Mining Attributed Input Grammars and their Applications in Fuzzing,” in 2023 IEEE Conference on Software Testing, Verification and Validation (ICST). Dublin, Ireland: IEEE, Apr. 2023, pp. 493–495

Show all 17 references
  1. [9]

    Generating Inputs for Grammar Mining using Dynamic Symbolic Execution,

    A. Pointner, J. Pichler, and H. Pr ¨ahofer, “Generating Inputs for Grammar Mining using Dynamic Symbolic Execution,” The Art, Science, and Engineering of Programming , vol. 10, no. 2, Jun. 2025, publisher: Aspect-Oriented Software Association (AOSA)

  2. [10]

    Synthesis of Semantic Actions in Attribute Grammars,

    P. K. Kalita, M. J. Kumar, and S. Roy, “Synthesis of Semantic Actions in Attribute Grammars,” Oct. 2022. [Online]. Available: https://repositum.tuwien.at/handle/20.500.12708/81366

  3. [11]

    Towards synthesizing complex programs from input-output examples,

    X. Chen, C. Liu, and D. Song, “Towards synthesizing complex programs from input-output examples,” arXiv preprint arXiv:1706.01284 , 2017

  4. [12]

    Input invariants,

    D. Steinh ¨ofel and A. Zeller, “Input invariants,” in Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering . Singapore Singapore: ACM, Nov. 2022, pp. 583–594. [Online]. Available: https://dl.acm.org...

  5. [13]

    From Grammar Inference to Semantic Inference—An Evolutionary Approach,

    Z. Kova ˘cevi´c, M. Mernik, M. Ravber, and M. ˇCrepin˘sek, “From Grammar Inference to Semantic Inference—An Evolutionary Approach,” Mathematics, vol. 8, no. 5, p. 816, May 2020. [Online]. Available: https://www.mdpi.com/2227-7390/8/5/816

  6. [14]

    Mining constraints for grammar fuzzing,

    M. Mera, “Mining constraints for grammar fuzzing,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis. Beijing China: ACM, Jul. 2019, pp. 415–418. [Online]. Available: https://dl.acm.org/doi/10.1145/3293882.3338983

  7. [15]

    The fuzzing book,

    A. Zeller, R. Gopinath, M. B ¨ohme, G. Fraser, and C. Holler, “The fuzzing book,” 2019. [Online]. Available: https://www.fuzzingbook.org/

  8. [16]

    Grammarinator: a grammar-based open source fuzzer,

    R. Hod ´ov´an, A. Kiss, and T. Gyim ´othy, “Grammarinator: a grammar-based open source fuzzer,” in Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation . Lake Buena Vista FL USA: ACM, Nov. 2018, pp. 45–48, unread. ...

  9. [17]

    A. V . Aho, M. S. Lam, R. Sethi, and J. D. Ullman, Compilers: principles, techniques, and tools , 2nd ed., ser. Pearson custom library. Essex: Pearson, 2014

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.