From Legal Text to Tech Specs: Generative AI's Interpretation of Consent in Privacy Law

Aniket Kesari; Anmol Singhal; Sarah Santos; Tom Norton; Travis Breaux

REVIEW 4 major objections 5 minor 39 references

From Legal Text to Tech Specs: Generative AI's Interpretation of Consent in Privacy Law

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs can partially automate CCPA compliance checking, but their reasoning is unreliable: chain-of-thought prompting yields 66.7% accuracy on use-case selection, and only 2 of 12 generated modifications are both non-violative and…

desk verdict Honest small study with a useful warning about LLMs and legal compliance, undercut by single-rater labels and a redacted citation. read the letter →

arxiv 2507.04185 v1 pith:B2B53U4D submitted 2025-07-05 cs.SE

Aniket Kesari , Travis Breaux , Tom Norton , Sarah Santos , Anmol Singhal This is my paper

classification cs.SE

keywords CCPAconsentcompliancelargelanguagemodelsusecaseanalysischain-of-thoughtpromptingrequirementsengineeringprivacylawprogram-aided

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a large language model can help translate CCPA consent requirements into software engineering checks, by classifying mobile-app use cases as needing modification and proposing concrete edits. On a hand-built set of 30 use cases, GPT-4o with chain-of-thought prompting correctly flagged 66.7% of cases needing compliance review versus 53.3% with plain Yes/No prompting. Of 12 modifications the model generated, 8 were judged not to violate the legal provision, but only 2 were internally consistent. The authors conclude that current LLMs can do preliminary screening but not final compliance decisions. If right, this matters because consent obligations in privacy law must be operationalized in code, and generative tools are entering that workflow.

What carries the argument

The carrying mechanism is a three-step pipeline: (1) prompt the LLM to turn app-store descriptions into use cases with preconditions, flow, and postconditions; (2) classify each use case as needing modification using either a zero-shot Yes/No prompt or a zero-shot chain-of-thought prompt that produces a rationale before the answer; (3) for flagged cases, use a program-aided language (PAL) prompt asking the model to output a Python program that rewrites the JSON-formatted use case. The evaluation machinery consists of ground-truth labels from a single legal expert, two manual criteria (non-violativity: the edit does not violate the legal requirement; self-consistency: the edit agrees with the use case's preconditions and postconditions), and syntactic similarity scores BLEU, ROUGE-1, and ROUGE-L comparing model edits to ground-truth change lists.

What would settle it

Re-run the 30-use-case selection task with several independent legal experts labeling the same cases; if their Yes/No answers disagree on any substantial fraction of cases, the reported accuracies are measuring agreement with one labeler, not legal compliance.

Watch

Extended reading notes

Core claim

The central claim is that LLMs can partially automate CCPA consent-compliance checking and modification, yet their legal reasoning is too unreliable to decide compliance. The paper demonstrates this through a three-step pipeline on real app-store-derived use cases: the model selects candidate non-compliant use cases, writes a Python program that edits the JSON-structured use case, and the authors manually judge the result. Chain-of-thought prompting improves selection accuracy over zero-shot Yes/No prompting, but both remain below what would justify automated decisions; the generated modifications are frequently non-violative yet inconsistent with the use case's own preconditions and postconditions. The authors' stated position is that generative AI currently lacks the ability to capture the nuances of consent requirements and should be used to flag evidence for legal experts, not to render compliance verdicts.

Load-bearing premise

The reported accuracies assume the single legal expert's Yes/No labels, on which all accuracy and modification judgments are based, are correct and repeatable.

Editorial extensions

If this is right

Chain-of-thought prompting is likely to become the default for compliance triage, since it improved selection accuracy from 53.3% to 66.7%.
LLM-based compliance tools should be scoped to pre-screening: they can point reviewers at evidence of possibly non-compliant behavior, but should not issue compliance decisions.
Compliance edits should be checked for self-consistency as a separate concern; the paper finds that most edits that are legally acceptable still break the use case's internal logic.
Automated review of generated modifications can be improved by unit-test-like heuristics derived from a reference set of legally compliant use cases, replacing some manual judgment.
Regulators and agencies such as the CPPA can reduce the gap by issuing engineering-oriented guidance, since even clearly written provisions leave implementation questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the ground truth comes from a single legal expert, the reported accuracy may be closer to agreement with one labeler than to legal compliance; a multi-expert replication could move the numbers substantially.
The same pipeline applied to GDPR-style standards such as freely given, specific, informed and unambiguous would likely perform worse, since those provisions are less rule-like than the two CCPA opt-in provisions tested.
The use cases are generated from app-store descriptions, so the pipeline tests the LLM's reading of marketing text rather than actual system behavior; real compliance checking would need access to internal requirements.
A concrete extension would be to have the LLM generate multiple alternative compliant edits per use case and measure how many distinct compliant states are reached, rather than judging a single edit.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Kesari et al. report a small, incremental study: GPT-4o applied to 30 hand-picked use cases for two CCPA provisions, with zero-shot and CoT classification followed by program-aided modification. The headline 66.7% CoT accuracy is not the story; the story is that the model's reasoning is fragile and that the authors are candid about it. They show concrete examples where the model ignores a pre-condition that the user already opted in, and they conclude that LLMs can only pre-screen, not decide. That conclusion holds up.

What is genuinely new is the CCPA focus; prior work was mostly GDPR. The pipeline is described clearly, the prompts are given, and the authors acknowledge the subjectivity of legal interpretation and the limits of BLEU/ROUGE. That is good practice for a preliminary study.

The soft spots are real and mostly in the evaluation. The ground truth comes from a single labeler, with no inter-rater reliability check and no codebook. The stress-test note is right: if another qualified lawyer labels a few cases differently, the 66.7% and the 8/12 and 2/8 counts shift. The sample of 30 gives no confidence intervals and no test for whether the CoT improvement over zero-shot is anything beyond noise. The ground-truth change lists used for BLEU/ROUGE were written by the same team, which is circular in a mild way. And the redacted citation to the prior GDPR paper is a genuine problem - readers cannot verify the novelty claim or build on the earlier method. The data and prompts are not released either.

None of this sinks the central qualitative claim. The paper is a preliminary benchmark, and the authors say so. The value is in the failure cases and the warning to anyone building compliance tools on raw LLM output. That is worth publishing as a short paper, but only if the authors address the labeler issue with at least one additional rater or at least report disagreement rates, and either release the data or explain why they cannot.

I would send this to a serious referee. It is not a desk reject. The math is simple accuracy, and there is no hidden fitting. The thinking is clear and honest. It just needs a stronger evaluation.

For the reading group: maybe, if you want to discuss evaluation standards for LLM-as-judge studies.

Referee Report

4 major / 5 minor

Summary. The manuscript proposes a three-step pipeline for using GPT-4o to support CCPA consent-compliance checking of software use cases: generating use cases from mobile app descriptions, classifying use cases as requiring modification under the CCPA opt-in provision, and generating Python-based modifications for flagged use cases. The authors report 53.3% accuracy with zero-shot Yes/No prompting and 66.7% with chain-of-thought prompting on 30 use cases, and, for the 12 cases passed to modification, 8/12 non-violative and 2/8 self-consistent modifications. The paper's qualitative conclusion is that LLMs can assist as pre-screening tools but are not reliable legal reasoners.

Significance. If the results hold, the paper provides a useful empirical pilot for evaluating LLM-based privacy-compliance tools and a concrete argument that such tools require human-in-the-loop review. The main strengths are the use of real app-store descriptions, transparent prompt templates, explicit modification-operation criteria, and an honest acknowledgment of LLM reasoning limitations. The significance is limited by the small sample size and by the reliance on a single legal expert for ground-truth labeling; the paper is best read as a preliminary benchmark rather than a settled quantitative evaluation. It does not claim parameter-free derivations, reproducible artifacts, or machine-checked proofs, and none are provided.

major comments (4)

[§4.4, §5.1, §5.2] The reported accuracy, non-violativity, and self-consistency figures all depend on ground-truth labels produced by a single legal expert ('the labeler'), with the change lists in §4.4 itemized by two authors of the paper and with no inter-rater reliability statistic or codebook. Because the relevant CCPA Regulations (Cal. Code Regs. tit. 11, §§ 7028(a) and 7071) leave room for interpretive choice, a second independent legal expert could plausibly label several of the 30 instances differently, which would shift the 66.7% CoT accuracy, the 8/12 non-violative count, and the 2/8 self-consistent count. The manuscript's own §6.2 acknowledges 'the subjective nature of legal interpretation,' so the concern is not disagreement with consensus but the absence of a measurement of labeling reliability. Add a second rater and report inter-rater agreement (e.g., Cohen's kappa) and per-item disagreements; if a second rater is infeasible, make the labeler's legal interpretations explicit as a codebook.
[§5.1, Table 3] The claim that CoT prompting improves accuracy by 13.4 percentage points is based on 30 binary responses (20 vs. 16 correct) with no confidence intervals, no paired significance test, and no effect size. With n=30, the exact binomial 95% confidence interval for 20/30 spans roughly 45–83%, so the difference between 16 and 20 is not clearly reliable. Report binomial confidence intervals for each accuracy and apply a paired test such as McNemar's test; if the data do not support a significance claim, present the result as descriptive.
[§5.2] The modification-stage reporting is ambiguous about the denominator. The paper says 'based on the ground truth labels, eight of the 12 predictions were correct' and then 'eight out of the 12 instances were non-violative,' without clarifying whether the non-violative set is the same eight true positives or includes some of the four false positives. Since the four false positives are deliberately passed to the modification stage, the end-to-end behavior should be reported as a confusion-matrix-style breakdown: among the eight true positives, how many generated modifications are non-violative and self-consistent, and among the four false positives, whether the modifications are redundant or introduce new violations.
[§5.2, Table 4] The BLEU/ROUGE comparison rests on very small and unbalanced groups: if the interpretation of the previous comment is correct, the first row has only two instances and the second row has six. The paper reports a single score per row with no per-instance values or variability, so the claim that self-consistency is associated with higher syntactic similarity is not supported. Report per-instance similarity scores and the number of instances in each row, and either present the scores as descriptive or drop the comparative claim.

minor comments (5)

[Table 4] The header spells 'BLEU' as 'BLUE'; this should be corrected.
[§4.1] The paper says 'we hand-selected 30 use cases' without stating the selection criteria; a brief description of the sampling or selection procedure would improve reproducibility.
[§1] The introduction refers to 'Part 2' through 'Part 7' while the body uses 'Section'; unify the terminology.
[§1] The sentence 'developers are beginning to use use LLMs to generate and critique requirements' contains a doubled word ('use use').
[§6.2] The phrase 'To study improve upon qualitative metrics' is ungrammatical and should be rewritten; nearby, 'we foresee a opportunities' is also a typo.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the evaluation is an empirical benchmark against human labels, not a derivation from fitted inputs.

full rationale

This paper performs an empirical evaluation of LLM performance on CCPA consent-compliance tasks, rather than a derivation or fitting exercise. The central quantitative results—66.7% CoT selection accuracy, 8/12 non-violative modifications, and 2/8 self-consistent modifications—are computed by comparing LLM outputs to human judgments. These human labels are not derived from the model's outputs, and no parameter is fitted to the test data. The ground-truth change lists used for BLEU/ROUGE comparison are manually itemized by the authors from the legal provisions, independently of the LLM-generated changes. There is no equation in which a predicted quantity equals an input by construction, and no fitted parameter is renamed as a prediction. The redacted citation to prior GDPR work is motivational context, not a load-bearing justification for the empirical findings. The acknowledged subjectivity of legal interpretation and the reliance on a single labeler are genuine threats to external validity, but they are evaluation-reliability concerns, not circularity. Even if another legal expert would label some instances differently, that would shift accuracy figures without making the evaluation circular. No self-citation chain is used to forbid alternatives or to justify the core result. Accordingly, no circular step can be identified with the required specificity, and the paper receives a score of 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on model-generated use cases standing in for real software, on a single legal expert's subjective labels as ground truth, and on the accuracy of the quoted CCPA provisions. No free parameters are fitted; this is not a curve-fitting paper.

assumptions (4)

domain assumption LLM-generated use cases from public app store descriptions are representative of real-world software use cases.
Section 4.1 generates use cases via GPT-4o from app descriptions instead of using proprietary company use cases. All downstream accuracy claims depend on this representativeness.
domain assumption The single legal expert's manual labels are the correct ground truth for CCPA compliance.
Section 4.4 defines accuracy against one labeler with no inter-rater reliability. Subjective legal interpretation makes this a load-bearing assumption.
domain assumption The quoted CCPA provisions, the two-step opt-in rule and the special rules for minors, are accurately summarized and are the relevant legal obligations.
These provisions are used in the prompts and in ground-truth labeling. Any misreading propagates to all classifications and modifications.
domain assumption GPT-4o, version gpt-4o-2024-05-13, is an adequate representative of LLM behavior for this task.
Section 4 uses a single model at temperature zero. Conclusions about LLMs in general are extrapolated from this one model.

how reviews work

0 comments

Cite this review

Pith. "Pith review of From Legal Text to Tech Specs: Generative AI's Interpretation of Consent in Privacy Law." pith.science (2026). https://pith.science/paper/B2B53U4D

@misc{pith2026250704185,
  author       = {Pith},
  title        = {Pith review of: From Legal Text to Tech Specs: Generative AI's Interpretation of Consent in Privacy Law},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B2B53U4D}},
  note         = {Machine review of arXiv:2507.04185}
}

read the original abstract

Privacy law and regulation have turned to "consent" as the legitimate basis for collecting and processing individuals' data. As governments have rushed to enshrine consent requirements in their privacy laws, such as the California Consumer Privacy Act (CCPA), significant challenges remain in understanding how these legal mandates are operationalized in software. The opaque nature of software development processes further complicates this translation. To address this, we explore the use of Large Language Models (LLMs) in requirements engineering to bridge the gap between legal requirements and technical implementation. This study employs a three-step pipeline that involves using an LLM to classify software use cases for compliance, generating LLM modifications for non-compliant cases, and manually validating these changes against legal standards. Our preliminary findings highlight the potential of LLMs in automating compliance tasks, while also revealing limitations in their reasoning capabilities. By benchmarking LLMs against real-world use cases, this research provides insights into leveraging AI-driven solutions to enhance legal compliance of software.

Figures

Figures reproduced from arXiv: 2507.04185 by the authors.

**Figure 1.** Criteria for use case modification • insert_post(condition) • remove_post(condition) The ground truth change list is used to compare changes with those generated by the LLM. We manually label the modified use cases using two dimensions that together define overall correctness: non-violativity, which is the labeler’s determination that the change creates a state in the system that does not violate the meaning of a le… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

39 extracted references · 19 canonical work pages

[1]

Mousumi Akter, Naman Bansal, and Shubhra Kanti Karmaker. 2022. Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE?. In Findings of the Association for Computational Linguistics: ACL 2022 , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, ...

work page 2022
[2]

Frank Armour and Granville Miller. 2001. Advanced Use Case Modeling: Software Systems (1st edition ed.). Addison-Wesley Professional, Boston

work page 2001
[3]

Chetan Arora, John Grundy, and Mohamed Abdelrazek. 2024. Advancing re- quirements engineering through generative ai: Assessing the role of llms. , 129–148 pages

work page 2024
[4]

Grounded Copilot: How Programmers Interact with Code- Generating Models

Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Replication Package for Article: "Grounded Copilot: How Programmers Interact with Code- Generating Models" 7, OOPSLA1 (April 2023), 78:85–78:111. https://doi.org/10. 1145/3586030

work page 2023
[5]

Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of Bleu in Machine Translation Research. In 11th Conference of the European Chapter of the Association for Computational Linguistics , Diana McCarthy and Shuly Wintner (Eds.). Association for Computational Linguistics, Trento, Italy, 249–256. https://aclanthology.org/E06-1032

work page 2006
[6]

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. CodeT: Code Generation with Generated Tests. https://doi.org/10.48550/arXiv.2207.10397 arXiv:2207.10397 [cs]

arXiv doi:10.48550/arxiv.2207.10397 2022
[7]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv doi:10.48550/arxiv.2107.03374 2021
[8]

Arman Cohan and Nazli Goharian. 2016. Revisiting Summarization Evaluation for Scientific Articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, ...

work page 2016

Show all 39 references

[9]

Tobias Eisenreich, Sandro Speth, and Stefan Wagner. 2024. From requirements to architecture: an AI-based journey to semi-automatically generate software architectures. , 52–55 pages

2024
[10]

Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin
[11]

Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y. Wu, Y. K. Li, Fuli Luo, Yingfei Xiong, and Wenfeng Liang. 2024. DeepSeek-Coder: When the Large Language Model Meets Program- ming – The Rise of Code Intelligence. https://doi.org/1...

2024 doi arXiv
[12]

Perttu Haemalaeinen, Mikke Tavast, and Anton Kunnari. 2023. Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for C...

2023
[13]

Ellen Jiang, Edwin Toh, Alejandra Molina, Kristen Olson, Claire Kayacik, Aaron Donsbach, Carrie J Cai, and Michael Terry. 2022. Discovering the Syntax and Strategies of Natural Language Programming with Generative Language Models. In Proceedings of the 2022 CHI Conference on H...

2022
[14]

Johnson, Daniel Tarlow, and Christian Walder

Daniel D. Johnson, Daniel Tarlow, and Christian Walder. 2023. R-U-SURE? Uncertainty-Aware Code Suggestions By Maximizing Utility Across Random User Intents. https://doi.org/10.48550/arXiv.2303.00732 arXiv:2303.00732 [cs]

2023 doi arXiv
[15]

Anubha Kabra, Sanketh Rangreji, Yash Mathur, Aman Madaan, Emmy Liu, and Graham Neubig. 2023. Program-Aided Reasoners (better) Know What They Know. https://doi.org/10.48550/arXiv.2311.09553 arXiv:2311.09553 [cs]

2023 doi arXiv
[16]

Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, Qian Liu, Evgenii Zheltonozhskii, Terry Yue Zhuo, Thomas Wang, Olivier Dehaene, Mishig Davaadorj, Joel Lamy-Poirier, João Monteiro, ...

2023 doi arXiv
[17]

Mankowitz, Esme Sutherland Robson, Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals

Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Ale...

2022 arXiv
[18]

Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng. 2023. Code as Policies: Language Model Programs for Embodied Control. https://doi.org/10.48550/arXiv.2209.07753 arXiv:2209.07753 [cs]

2023 doi arXiv
[19]

Chin-Yew Lin. 2004. ROUGE: A Package for Automatic Evaluation of Summaries. In Text Summarization Branches Out. Association for Computational Linguistics, Barcelona, Spain, 74–81. https://aclanthology.org/W04-1013

2004
[20]

Qianou Ma, Weirui Peng, Hua Shen, Kenneth Koedinger, and Tongshuang Wu
[21]

Giuseppe Marra, Sebastijan Dumančić, Robin Manhaeve, and Luc De Raedt. 2024. From Statistical Relational to Neurosymbolic Artificial Intelligence: a Survey. https://doi.org/10.48550/arXiv.2108.11451 arXiv:2108.11451 [cs]

2024 doi arXiv
[22]

Ani Nenkova. 2006. Summarization evaluation for text and speech: issues and approaches. In Interspeech. Proc. Interspeech 2006, Pittsburgh, PA, USA, paper

2006
[23]

Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023. Logic- LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoning. https://doi.org/10.48550/arXiv.2305.12295 arXiv:2305.12295 [cs]

2023 doi arXiv
[24]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a Method for Automatic Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , Pierre Isabelle, Eugene Charniak, and Dekang Lin (Eds...

2002
[25]

Gordon, Carina Negreanu, Christian Poelitz, Sruti Srini- vasa Ragavan, and Ben Zorn

Advait Sarkar, Andrew D. Gordon, Carina Negreanu, Christian Poelitz, Sruti Srini- vasa Ragavan, and Ben Zorn. 2022. What is it like to program with artificial intelligence? https://doi.org/10.48550/arXiv.2208.06213 arXiv:2208.06213 [cs]

2022 doi arXiv
[26]

Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2023. An empirical evaluation of using large language models for automated unit test generation

2023
[27]

Freda Shi, Daniel Fried, Marjan Ghazvininejad, Luke Zettlemoyer, and Sida I. Wang. 2022. Natural Language to Code Translation with Execution. https: //doi.org/10.48550/arXiv.2204.11454 arXiv:2204.11454 [cs]

2022 doi arXiv
[28]

Clive Thompson. 2022. How an AI Became My Code-Writing Genie. https: //www.wired.com/story/openai-copilot-autocomplete-for-code/ Section: tags

2022
[29]

Joe H. Ward. 1963. Hierarchical Grouping to Optimize an Objective Function. J. Amer. Statist. Assoc. 58, 301 (1963), 236–244. https://doi.org/10.2307/2282967 Publisher: [American Statistical Association, Taylor & Francis, Ltd.]

1963 doi
[30]

Jiayi Wei, Greg Durrett, and Isil Dillig. 2024. Coeditor: Leveraging Contextual Changes for Multi-round Code Auto-editing. https://doi.org/10.48550/arXiv. 2305.18584 arXiv:2305.18584 [cs]

2024 doi arXiv
[31]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. https://doi.org/10.48550/arXiv.2201.11903 arXiv:2201.11903 [cs]

2023 doi arXiv
[32]

Xu, Bogdan Vasilescu, and Graham Neubig

Frank F. Xu, Bogdan Vasilescu, and Graham Neubig. 2021. In-IDE Code Genera- tion from Natural Language: Promise and Challenges. https://doi.org/10.48550/ arXiv.2101.11149 arXiv:2101.11149 [cs]

2021 doi arXiv
[33]

Lewis, Chris- tian Kästner, and Tongshuang Wu

Chenyang Yang, Rishabh Rustogi, Rachel Brower-Sinning, Grace A. Lewis, Chris- tian Kästner, and Tongshuang Wu. 2023. Beyond Testers’ Biases: Guiding Model Testing with Knowledge Bases using LLMs. https://doi.org/10.48550/arXiv.2310. 09668 arXiv:2310.09668 [cs]

2023 doi arXiv
[34]

Ming Zhang, Chengzhang Li, Meilin Wan, Xuejun Zhang, and Qingwei Zhao
[35]

Xishuo Zhang, Lin Liu, Yi Wang, Xiao Liu, Hailong Wang, Anqi Ren, and Chetan Arora. 2023. PersonaGen: A Tool for Generating Personas from User Feedback . In 2023 IEEE 31st International Requirements Engineering Conference (RE) . IEEE Computer Society, Los Alamitos, CA, USA, 35...

2023
[38]

Expert Systems with Applications 237 (March 2024), 121364

ROUGE-SEM: Better evaluation of summarization using ROUGE combined with semantics. Expert Systems with Applications 237 (March 2024), 121364. https://doi.org/10.1016/j.eswa.2023.121364

2024
[2023]

2023), 111741

Out of the BLEU: How should we assess quality of the Code Generation models? Journal of Systems and Software 203 (Sept. 2023), 111741. https://doi. org/10.1016/j.jss.2023.111741

2023
[2024]

https://doi.org/10.48550/arXiv.2409.08775 arXiv:2409.08775 [cs]

What You Say = What You Want? Teaching Humans to Articulate Require- ments for LLMs. https://doi.org/10.48550/arXiv.2409.08775 arXiv:2409.08775 [cs]

doi arXiv
[2079]

https://doi.org/10.21437/Interspeech.2006-429

2006 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Mousumi Akter, Naman Bansal, and Shubhra Kanti Karmaker. 2022. Revisiting Automatic Evaluation of Extractive Summarization Task: Can We Do Better than ROUGE?. In Findings of the Association for Computational Linguistics: ACL 2022 , Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (Eds.). Association for Computational Linguistics, Dublin, Ireland, ...

work page 2022

[2] [2]

Frank Armour and Granville Miller. 2001. Advanced Use Case Modeling: Software Systems (1st edition ed.). Addison-Wesley Professional, Boston

work page 2001

[3] [3]

Chetan Arora, John Grundy, and Mohamed Abdelrazek. 2024. Advancing re- quirements engineering through generative ai: Assessing the role of llms. , 129–148 pages

work page 2024

[4] [4]

Grounded Copilot: How Programmers Interact with Code- Generating Models

Shraddha Barke, Michael B. James, and Nadia Polikarpova. 2023. Grounded Copilot: How Programmers Interact with Code-Generating Models. Replication Package for Article: "Grounded Copilot: How Programmers Interact with Code- Generating Models" 7, OOPSLA1 (April 2023), 78:85–78:111. https://doi.org/10. 1145/3586030

work page 2023

[5] [5]

Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the Role of Bleu in Machine Translation Research. In 11th Conference of the European Chapter of the Association for Computational Linguistics , Diana McCarthy and Shuly Wintner (Eds.). Association for Computational Linguistics, Trento, Italy, 249–256. https://aclanthology.org/E06-1032

work page 2006

[6] [6]

Bei Chen, Fengji Zhang, Anh Nguyen, Daoguang Zan, Zeqi Lin, Jian-Guang Lou, and Weizhu Chen. 2022. CodeT: Code Generation with Generated Tests. https://doi.org/10.48550/arXiv.2207.10397 arXiv:2207.10397 [cs]

arXiv doi:10.48550/arxiv.2207.10397 2022

[7] [7]

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

arXiv doi:10.48550/arxiv.2107.03374 2021

[8] [8]

Arman Cohan and Nazli Goharian. 2016. Revisiting Summarization Evaluation for Scientific Articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, ...

work page 2016