Pith. sign in

REVIEW 4 major objections 6 minor 43 references

Legal Requirements Translation from Law

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that prompting an LLM with a hand-built Python class hierarchy and similarity-selected demonstrations yields an executable code representation of legal provisions that passes 89.4% of unit tests and beats JSON extraction.

desk verdict A useful code-generation approach to legal requirements extraction whose reported accuracy is largely a measure of agreement with the authors' own metamodel. read the letter →

arxiv 2507.02846 v1 pith:2QLNXFFH submitted 2025-07-03 cs.SE cs.CL

Anmol Singhal , Travis Breaux This is my paper
classification cs.SEcs.CL
keywords legalrequirementtranslationtextualentailmentin-contextlearningmetadataextractioncodegenerationlargelanguagemodelsdatabreachnotificationlaws
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that legal provisions can be translated into a canonical, executable Python representation by a large language model, provided the model is constrained by a hand-built class structure and relevant examples. The representation preserves both the hierarchical layout of the law and the semantic relations among rules, conditions, references, and exemptions. Evaluated on 13 U.S. state data breach notification laws, the generated code passes 89.4% of all unit tests and reaches 82.2% precision and 88.7% recall, outperforming a JSON attribute-extraction baseline. If this holds, the method is a practical route to automated legal-requirements elicitation that needs only a handful of selected demonstrations rather than large labeled datasets.

What carries the argument

The load-bearing object is the Python metamodel: a class hierarchy with Section, Statement, Expression, Rule, Definition, Exemption, and Information classes, plus relation methods such as add_refines, add_exception, and add_follows that link statements through a Reference class. The second mechanism is the demonstration-selection pipeline: a zero-shot labeling prompt assigns metadata tags to the input paragraph, candidate demonstrations are filtered by label overlap, and the remaining candidates are ranked by cosine similarity of their embeddings, with the top three placed into the code-generation prompt. Together these components constrain the LLM to instantiate the metamodel rather than produce free-form text.

What would settle it

Have a second legal expert independently translate the same 182 test paragraphs into the Python class structure without seeing the original coding decisions, then score the generated code against that independent gold standard and report agreement between the two experts; if the scores drop materially or agreement is low, the reported accuracy is an artifact of annotation self-consistency.

Watch

Extended reading notes

Core claim

The central claim is that an LLM can produce a faithful, executable transcription of a legal paragraph when prompted with a closed Python class hierarchy that encodes legal metadata and their interrelationships, plus three demonstrations chosen by a textual-entailment label match and cosine similarity. Across 182 held-out paragraphs from six unseen state laws, the generated representations pass 89.4% of unit tests, with 82.2% precision and 88.7% recall, and a pass@3 score of 62.1%; the JSON baseline reaches only 31.2% pass@3. The paper interprets this as evidence that constraining generation to code preserves nested structure and cross-references better than flat attribute extraction, while requiring no model retraining.

Load-bearing premise

The load-bearing assumption is that the hand-written reference translations used as the test oracle are a complete and correct reading of the laws; if that reading is idiosyncratic, the high test-pass rates measure consistency with one annotator's interpretation rather than legal accuracy.

Editorial extensions

If this is right

  • A legal paragraph can be turned into executable objects that keep nested sections, cross-references, and exception and refinement relations explicit, which flat attribute lists lose.
  • The method transfers to laws not used in designing the prompts, since only a few demonstrations are sampled for each new provision instead of retraining a model.
  • The unit-testing scheme provides a reusable conformance check: generated representations can be scored programmatically against a predefined metamodel.
  • Strict whole-paragraph correctness is not achieved on a single run; pass@1 is about 40% and pass@3 about 62%, so practical use would sample multiple generations and select by test results.
  • Improvements are still needed for references, exemptions, and relationship types, which show the highest error rates and are the attributes most critical for legal exceptions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported accuracy measures agreement with one human's reference encoding, so a natural next experiment is to have a second annotator independently translate the same provisions and report inter-annotator agreement; the method's real ceiling is the fidelity of that coding frame.
  • The human effort has shifted rather than disappeared: someone must still design the class structure for each new legal domain through qualitative coding, so the promise of 'no large labeled datasets' applies to the LLM, not to the overall process.
  • A concrete extension is to run the same prompting pipeline with independently authored ground truth and an open-weight model, which would separate the contribution of the representation from model-specific behavior.
  • Downstream compliance tools could consume the serialized objects for traceability and conflict detection, but only for the attributes the metamodel captures; errors concentrated in references and relationship types would be the first barrier.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes an automated pipeline that translates legal provisions from U.S. state data breach notification statutes into Python code instantiating a manually designed class hierarchy (a metamodel). The pipeline uses GPT-4o with in-context learning, where demonstrations are selected by a zero-shot label-assignment step and cosine similarity on embeddings. The generated code is evaluated against a ground-truth corpus (332 paragraphs, 13 laws) via hand-authored unit tests covering compilation, structural attributes, and semantic attribute values. On a held-out test set of six laws (the paper also says seven), the method reports 89.4% semantic-test accuracy, 82.2% precision, 88.7% recall, and pass@3 of 62.1%, outperforming a JSON-based attribute extraction baseline, with ablations showing contributions from both the class structure and the demonstration-selection strategy.

Significance. If the reported accuracy reflects genuine legal-semantic fidelity, the approach would be a practical step toward semi-automated legal requirements elicitation, reducing the need for large labeled corpora and providing an executable, testable representation of regulations. The paper has concrete strengths: it makes code and data publicly available (Zenodo), includes a held-out test set, runs ablations, compares against a JSON baseline, and reports pass@k with a detailed unit-testing protocol. However, the significance is heavily tempered by the construction of the ground truth and metamodel by a single annotator (the first author), which makes the absolute accuracy numbers measures of self-consistency rather than validated legal interpretation. The central claim of a 'canonical' representation is therefore not yet established.

major comments (4)
  1. [Sections III-B, III-C, IV-A, and VII] The metamodel (Section III-B) and the ground truth translations (Section III-C) were both produced by the first author alone, and the unit tests in Section IV-A compare generated code against this same ground truth. Passing 89.4% of semantic tests therefore measures how well GPT-4o reproduces one annotator's coding decisions, not whether those decisions are legally accurate or canonical. Section VII acknowledges the risk but states only that the first author revisited the annotations. This is load-bearing because the abstract and conclusion describe the representation as 'canonical' and the method as producing semantically faithful translations. I recommend adding an independent validation component, such as a second annotator coding a sample of paragraphs with an inter-annotator agreement measure, or a legal expert reviewing a sample of ground-truth translations and the derived class structure. Without such evidence, the absolute accuracy and generalizability claims are not established.
  2. [Introduction, Section III-C, Section IV-B, and Section V] The paper contains inconsistent counts that affect reproducibility. The Introduction says the test set is 'seven unseen state regulations,' while Section III-C says the test set consists of 182 paragraphs 'corresponding to six laws,' and Section V also says 'across six legal texts.' Similarly, the Introduction and Section IV-B refer to '21 distinct test cases' or '21 syntactic and semantic test cases,' whereas Section IV-A states there are 22 tests (1 compilation + 5 structural + 16 semantic). These inconsistencies make it impossible for a reader to know the true test-set composition and the denominator used in the reported metrics. Please reconcile the numbers and ensure the abstract, method, and results sections agree.
  3. [Section IV-A (Semantic Tests)] The semantic tests use exact string match after normalization (lowercase, stop-word removal, punctuation stripping). Legal semantics are not always preserved by such normalization; for example, 'may be delayed' versus 'is delayed' could be semantically distinct but would be treated as a mismatch, while paraphrases with different stop words could match erroneously. Since semantic-test accuracy is a central reported metric (Table II), the choice of this strict comparison is load-bearing. The paper should either provide evidence that the normalization+exact-match procedure aligns with legal semantic equivalence (e.g., via a small human-judged sample) or report results using a more tolerant semantic similarity measure, and discuss how the strict criterion affects the reported precision/recall, particularly for attribute types like references and relationship types that have low precision/recall in Table III.
  4. [Section III-C and Title/Abstract] The title and abstract claim the approach is based on 'textual entailment,' but the described demonstration-selection method is a zero-shot label-assignment prompt (Figure 4) followed by exact matching of predicted labels and cosine similarity using text embeddings. No textual entailment model or natural language inference component is used anywhere in the pipeline. If the authors intend 'textual entailment' as a loose term for semantic relatedness, this should be stated explicitly; otherwise, the method description is inaccurate. I recommend either renaming the contribution to something like 'label-based retrieval with embedding similarity' or providing a concrete entailment-based step (e.g., using an NLI model to score candidate demonstrations) and showing its impact on performance.
minor comments (6)
  1. [Section V (first paragraph after Table II)] The text says 'the structural test and semantic test accuracy of approximately 90%' but Table II reports structural test accuracy of 82.0% on the test set and semantic test accuracy of 89.4%. The wording is misleading and should be corrected to reference the specific values in Table II.
  2. [Section IV-B, Equation (1)] Because the experiments set n = k, the pass@k formula in Equation (1) reduces to the simple fraction c/n, not the unbiased estimator typically used for pass@k (where n > k is required). Please clarify that the reported pass@k is the empirical pass rate for k samples, or use the standard estimator with n larger than k.
  3. [Section VI-A] The error analysis states that 'we identified 20 unique errors' but does not provide the error list, examples, or a taxonomy table. Including the actual errors and representative examples would make the root-cause analysis reproducible and more convincing.
  4. [Table III] The 'Relationship type' row aggregates six relationship types into one precision/recall score. Because the aggregate hides large variation (e.g., some relationship types may be much harder than others), reporting per-type results would be more informative.
  5. [Figure 3 (lines 10 and 35)] In the code example, 'r1.add_is_exception_to(ref1)' with ref1 targeting r1 itself is confusing: the reference object points to the very rule it is an exception to. Please clarify the intended semantics of the 'target' field in Reference and illustrate with a less self-referential example.
  6. [References] References [15] and [32] are the same arXiv paper (Chen et al., 'Evaluating Large Language Models Trained on Code') and are cited with different numbers. Please merge or disambiguate them.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation compares generated code to a human-authored ground truth; the acknowledged single-annotator limitation is a validity concern, not a derivation-circular step.

full rationale

The paper's derivation chain is: (1) the first author performs open coding on the corpus to design a Python class metamodel (Section III-B); (2) the first author manually translates 332 paragraphs into this metamodel to create ground truth code (Section III-C); (3) GPT-4o is prompted with the class definitions and demonstrations selected by textual entailment and cosine similarity to generate code for each paragraph (Section III-D); and (4) unit tests compare the generated attribute values against the ground truth (Section IV-A). This is a standard supervised evaluation against a human-constructed gold standard: the LLM output is compared to held-out human annotations, not to the model's own inputs or to a fitted parameter. No equation in the paper reduces to another by construction, no fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing ansatz is imported from the authors' prior work. The self-citations (e.g., [1], [16], [21], [43]) are contextual background and do not carry the central argument. The paper itself acknowledges in Section VII that the ground truth relies on human annotations and that other researchers might define attributes differently; this is an external-validity limitation about the canonicality of the representation, not circularity in the derivation. The reported 89.4% accuracy measures how well GPT-4o reproduces one annotator's coding decisions, which is a meaningful reproducibility result but not proof of legal correctness; however, that does not make the derivation circular. No specific circular step can be exhibited, so the score is 0.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on the manually constructed Python metamodel and the first author's ground truth translations, both of which are pipeline inputs rather than independently verified facts. The list includes design choices that are neither fitted nor ablated. The axioms are the domain assumptions about completeness, annotation correctness, metric validity, and sample independence. No new physical or conceptual entities are introduced; the Python classes are representational constructs defined by the authors.

free parameters (4)
  • In-context demonstration count = 3
    Section III-C selects three demonstrations per prompt; no sensitivity analysis is reported.
  • LLM sampling temperature = 0.5
    Section III-D sets temperature to 0.5; no ablation on temperature is reported.
  • Pass@k sampling design = n = k for k = 1 to 5
    Section IV-B computes pass@k with n=k rather than the standard n>k estimator, which changes the variance and interpretation of the pass@k scores.
  • Dev/test split saturation point = 150 dev paragraphs / 182 test paragraphs
    Section III-C defines the dev set as paragraphs coded before saturation; this split determines which laws are 'unseen' and affects generalization claims.
assumptions (5)
  • domain assumption The Python class structure is a complete canonical representation of the metadata in the 13 data breach laws.
    Section III-B: derived by first author open coding until saturation; not validated externally.
  • domain assumption The first author's ground truth annotations are correct and consistent.
    Section IV-A and Section VII: single annotator; no inter-rater reliability.
  • domain assumption Exact match after normalization is a sufficient proxy for semantic equivalence of extracted attributes.
    Section IV-A: semantic tests use normalized exact string matching; legal paraphrases may be counted as errors.
  • standard math Pass@k formula assumes independent, identically distributed model samples.
    Section IV-B: the binomial combination formula is standard but assumes sample independence.
  • domain assumption LLM in-context learning with embedding-based retrieval generalizes to unseen paragraphs in the same legal domain.
    Section III-C and Section VI-A: retrieval errors are a known source of failure; generalization to other domains is untested.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Legal Requirements Translation from Law." pith.science (2026). https://pith.science/paper/2QLNXFFH

@misc{pith2026250702846,
  author       = {Pith},
  title        = {Pith review of: Legal Requirements Translation from Law},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2QLNXFFH}},
  note         = {Machine review of arXiv:2507.02846}
}
read the original abstract

Software systems must comply with legal regulations, which is a resource-intensive task, particularly for small organizations and startups lacking dedicated legal expertise. Extracting metadata from regulations to elicit legal requirements for software is a critical step to ensure compliance. However, it is a cumbersome task due to the length and complex nature of legal text. Although prior work has pursued automated methods for extracting structural and semantic metadata from legal text, key limitations remain: they do not consider the interplay and interrelationships among attributes associated with these metadata types, and they rely on manual labeling or heuristic-driven machine learning, which does not generalize well to new documents. In this paper, we introduce an approach based on textual entailment and in-context learning for automatically generating a canonical representation of legal text, encodable and executable as Python code. Our representation is instantiated from a manually designed Python class structure that serves as a domain-specific metamodel, capturing both structural and semantic legal metadata and their interrelationships. This design choice reduces the need for large, manually labeled datasets and enhances applicability to unseen legislation. We evaluate our approach on 13 U.S. state data breach notification laws, demonstrating that our generated representations pass approximately 89.4% of test cases and achieve a precision and recall of 82.2 and 88.7, respectively.

Figures

Figures reproduced from arXiv: 2507.02846 by the authors.

Figure 1
Figure 1. Maryland Personal Information Protection Act (§14–3504) [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. UML Class Diagram for Python Code Structure [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Structured Representation of Section 14–3504 in Code Form [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Prompt to Label Legal Text After generating labels for the input text paragraph, we retrieve demonstrations from the development set with the highest score, assigning one point for each matching label. From this list, we select three demonstrations with the highest cos…
Figure 5
Figure 5. Figure 5: Prompt to Translate Legal Text into Python [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Baseline Prompt for Direct Attribute Extraction [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Pass@k score trend on the test set including the class structure and demonstration strategy, in response to RQ3 that asks “To what extent do the method steps contribute to overall performance improvement?”. The pass@k score trend, which denotes the number of paragraphs…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 37 canonical work pages

  1. [1]

    Towards regulatory compliance: Extracting rights and obligations to align requirements with regulations

    T.D. Breaux, M.W. Vail, A.I. Anton. (2006). “Towards regulatory compliance: Extracting rights and obligations to align requirements with regulations.” 14th IEEE International Requirements Engineering Conference, pp. 49-58

    work page 2006

  2. [2]

    Language Models are Few-Shot Learners,

    Brown et al., “Language Models are Few-Shot Learners,” Advances in Neural Information Processing Systems (NeurIPS) , 33, 2020

    work page 2020

  3. [3]

    Rangreji, Y

    A, Kabra, S. Rangreji, Y . Mathur, A. Madaan, E. Liu, and G. Neubig

    work page

  4. [4]

    Logical consistency of large language models in fact-checking

    B. Ghosh, S. Hasan, N.A. Arafat, A. Khan. (2025). “Logical consistency of large language models in fact-checking.” International Conference on Learning Representations

    work page 2025

  5. [5]

    Consistency analysis of ChatGPT

    M. Jang, T. Lukasiewicz. (2023). “Consistency analysis of ChatGPT.” Empirical Methods in Natural Language Processing , pp. 15970–15985

    work page 2023

  6. [6]

    Maieutic prompting: Logically consistent reasoning with recursive explanations

    J. Jung, L. Qin, S. Welleck, F. Brahman, C. Bhagavatula, R. Le Bras, Y . Choi. (2022). “Maieutic prompting: Logically consistent reasoning with recursive explanations.” Empirical Methods in Natural Language Processing, pp. 1266–1279

    work page 2022

  7. [7]

    Large Language Models are Zero-shot Reasoners

    T. Kojima, S. Gu, M. Reid, Y . Matsuo, Y . Iwasawa. “Large Language Models are Zero-shot Reasoners.” Advances in Neural Information Processing Systems (NeurIPS) 35, pp. 22199-22213, 2022

    work page 2022

  8. [8]

    Lost in the Middle: How Language Models Use Long Contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, P. Liang. “Lost in the Middle: How Language Models Use Long Contexts,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 157–173, 2024

    work page 2024

Show all 43 references
  1. [9]

    Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity,

    Y . Lu, M. Bartolo, A. Moore, S. Riedel, P. Stenetorp, “Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity,” 60th Annual Meeting of the Association for Computational Linguistics, pp. 8086–8098, 2022

    2022

  2. [10]

    A legal cross-references taxonomy for identifying conflicting software requirements

    J.C. Maxwell, A.I. Antón, P. Swire, P. (2011). “A legal cross-references taxonomy for identifying conflicting software requirements.” 19th IEEE International Requirements Engineering Conference , pp. 197-206

    2011

  3. [11]

    Enhancing self-consistency and performance of pre- trained language models through natural language inference

    E. Mitchell, J. Noh, S. Li, W. Armstrong, A. Agarwal, P. Liu, C. Finn, C. Manning. (2022). “Enhancing self-consistency and performance of pre- trained language models through natural language inference.” Empirical Methods in Natural Language Processing , pp. 1754–1768

    2022

  4. [12]

    On Context Utilization in Summarization with Large Language Models,

    M. Ravaut, A. Sun, N. Chen, and S. Joty. “On Context Utilization in Summarization with Large Language Models,” 62 nd Annual Meeting of the Association for Computational Linguistics vol. 1, pp. 2764–2781, 2024

    2024

  5. [13]

    An automated framework for detection and resolution of cross references in legal texts

    N. Sannier, M. Adedjouma, M., Sabetzadeh, L. Briand (2017). “An automated framework for detection and resolution of cross references in legal texts.” Requirements Engineering, 22: 215-237

    2017

  6. [14]

    J. Saldanã. The Coding Manual for Qualitative Researchers, SAGE Publications, 2012

    2012

  7. [16]

    Regulatory Requirements Traceability and Analysis Using Semi-formal Specifications

    T.D. Breaux, D.G. Gordon. “Regulatory Requirements Traceability and Analysis Using Semi-formal Specifications”. Requirements Engineering: Foundation for Software Quality. REFSQ 2013. Lecture Notes in Com- puter Science, vol 7830. Springer, Berlin, Heidelberg, 2013

    2013

  8. [18]

    Legal requirements metrics for compliance analysis

    A Massey. “Legal requirements metrics for compliance analysis”. PhD thesis, North Carolina State University (2012)

    2012

  9. [19]

    Evaluating existing security and privacy requirements for legal compliance

    AK Massey, PN Otto, LJ Hayward, AI Anton. “Evaluating existing security and privacy requirements for legal compliance”. Requirements Engineering 15(1):119–137 (2010)

    2010

  10. [20]

    Akoma-Ntoso for Legal Documents

    M. Palmirani, F. Vitali. “Akoma-Ntoso for Legal Documents”. Leg- islative XML for the Semantic Web. Law, Governance and Technology Series, vol 4. Springer, Dordrecht (2011)

    2011

  11. [21]

    Requirements Satisfiability with In-Context Learning,

    S. Santos, T.D. Breaux, T. Norton, S. Haghighi, S. Ghanavati (2024). “Requirements Satisfiability with In-Context Learning,” International Requirements Engineering Conference

    2024

  12. [22]

    GaiusT: supporting the extraction of rights and obligations for regulatory compliance

    Zeni, N., Kiyavitskaya, N., Mich, L. et al. “GaiusT: supporting the extraction of rights and obligations for regulatory compliance”. Require- ments Engineering 20, 1–22 (2015)

    2015

  13. [23]

    Large-scale multi-label text classification on EU legislation

    Chalkidis, I., Fergadiotis, M., Malakasiotis, P., Androutsopoulos, I. (2019). “Large-scale multi-label text classification on EU legislation”. arXiv preprint arXiv:1906.02192

    2019 arXiv

  14. [24]

    An automated framework for the extraction of semantic legal metadata from legal texts

    Sleimi, A., Sannier, N., Sabetzadeh, M. et al. “An automated framework for the extraction of semantic legal metadata from legal texts”. Empir Software Eng 26, 43 (2021)

    2021

  15. [25]

    Populating legal ontologies using semantic role labeling

    Humphreys, L. and Boella, G. and van der Torre, L. and Robaldo, L. and Di Caro, L. and Ghanavati, S. and Muthuri, R. “Populating legal ontologies using semantic role labeling”, Artificial Intelligence and Law (2020)

    2020

  16. [26]

    Automated extraction of regulated information types using hyponymy relations

    Bhatia J, Evans MC, Wadkar S, Breaux TD (2016b) “Automated extraction of regulated information types using hyponymy relations”. Proceedings of the 3rd International Workshop on Artificial Intelligence for Requirements Engineering , pp 19–25

    2016

  17. [27]

    The production rule framework: devel- oping a canonical set of software requirements for compliance with law

    Maxwell JC, Anton AI (2010). “The production rule framework: devel- oping a canonical set of software requirements for compliance with law”. Proceedings of the ACM International Health Informatics Symposium , pp 629–636

    2010

  18. [28]

    NLP- Based Automated Compliance Checking of Data Processing Agreements Against GDPR

    Cejas, O.A., Azeem, M.I., Abualhaija, S., Briand, L.C. (2022). “NLP- Based Automated Compliance Checking of Data Processing Agreements Against GDPR”. IEEE Transactions on Software Engineering, 49, 4282- 4303

    2022

  19. [29]

    A survey on large language models for code generation

    Jiang, J., Wang, F., Shen, J., Kim, S., Kim, S. (2024). “A survey on large language models for code generation”. arXiv preprint arXiv:2406.00515

    2024 arXiv

  20. [30]

    Structured chain-of-thought prompt- ing for code generation

    Li, J., Li, G., Li, Y ., Jin, Z. (2025). “Structured chain-of-thought prompt- ing for code generation”. ACM Transactions on Software Engineering and Methodology, 34(2), 1-23

    2025

  21. [31]

    Li, Y ., Choi, D., Chung, J., Kushman, N., Schrittwieser, J., Leblond, R., ..., Vinyals, O. (2022). Competition-level code generation with alphacode. Science, 378(6624), 1092-1097

    2022

  22. [32]

    Evaluating large language models trained on code

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., ..., Zaremba, W. (2021). “Evaluating large language models trained on code”. arXiv preprint arXiv:2107.03374

    2021 arXiv

  23. [33]

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., ..., Sutton, C. (2021). Program synthesis with large language models. arXiv preprint arXiv:2108.07732

    2021 arXiv

  24. [34]

    Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models

    E. Jiang, E. Toh, A. Molina, K. Olson, C. Kayacik, A. Donsbach, C. J Cai, and M. Terry. 2022. “Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models”. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems

    2022

  25. [35]

    Ru-sure? uncertainty-aware code suggestions by maximizing utility across random user intents

    Johnson, D. D., Tarlow, D., and Walder, C. (2023). “Ru-sure? uncertainty-aware code suggestions by maximizing utility across random user intents”. arXiv preprint arXiv:2303.00732

    2023 arXiv

  26. [36]

    and Neubig, G

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y ., ... and Neubig, G. (2023, July). Pal: Program-aided language models. In Inter- national Conference on Machine Learning (pp. 10764-10799). PMLR

    2023

  27. [37]

    Case study research: Design and methods

    R. K. Yin, "Case study research: Design and methods", vol. 5. Sage , 2009

    2009

  28. [38]

    Towards Mitigating LLM Hallucination via Self Reflection

    Z. Ji, T. Yu, Y . Xu, N. Lee, E. Ishii, and P. Fung. 2023. “Towards Mitigating LLM Hallucination via Self Reflection”. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1827–1843, Singapore. Association for Computational Linguistics

    2023

  29. [39]

    Self-reflection in llm agents: Effects on problem-solving performance

    Renze, M., and Guven, E. (2024). “Self-reflection in llm agents: Effects on problem-solving performance.” arXiv preprint arXiv:2405.06682

    2024 arXiv

  30. [40]

    Addressing Legal Requirements in Require- ments Engineering

    P. N. Otto and A. I. Anton, “Addressing Legal Requirements in Require- ments Engineering.” 15th IEEE International Requirements Engineering Conference (RE 2007) , Delhi, India, 2007, pp. 5-14

    2007

  31. [41]

    Logic-Based Regulation Compliance- Assistance

    S. Kerrigan, K.H. Law. “Logic-Based Regulation Compliance- Assistance.” Proc. of the 9th Int’l Conf. on AI and Law , pp. 126-135, June 2003

    2003

  32. [42]

    Language Models of Code are Few-Shot Commonsense Learners

    A. Madaan, S. Zhou, U. Alon, Y . Yang, G. Neubig. “Language Models of Code are Few-Shot Commonsense Learners.” Proc. of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP 2022), pp. 1384–1403

    2022

  33. [43]

    Assessing regulatory change through legal requirements coverage modeling

    D. G. Gordon, T. D. Breaux (2013, July). “Assessing regulatory change through legal requirements coverage modeling”. In 2013 21st IEEE International Requirements Engineering Conference (RE) (pp. 145-154). IEEE

    2013

  34. [44]

    Legal Requirements Translation from Law

    A. Singhal, T. Breaux (2025). “Legal Requirements Translation from Law”. IEEE International Requirements Engineering Conference 2025 (RE), Valencia, Spain. Zenodo. https://doi.org/10.5281/zenodo.15794182

    2025 doi

  35. [2024]

    Program-Aided Reasoners (Better) Know What They Know

    “Program-Aided Reasoners (Better) Know What They Know”. Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2262–2278

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.