Pith. sign in

REVIEW 5 major objections 6 minor 73 references

Generating Privacy Stories From Software Documentation

T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs can turn software docs into privacy stories at F1 0.766

desk verdict A useful but small feasibility dataset for LLM-based privacy story generation, whose headline F1 claim is not supported by the paper's own tables and parsing rules. read the letter →

arxiv 2506.23014 v1 pith:4O4I52MM submitted 2025-06-28 cs.SE cs.AI

Wilder Baldwin , Shashank Chintakuntla , Shreyah Parajuli , Ali Pourghasemi , Ryan Shanz , Sepideh Ghanavati This is my paper
classification cs.SEcs.AI
keywords privacyrequirementslargelanguagemodelstaxonomyuserstorieschain-of-thoughtpromptingin-contextlearningsupervisedfine-tuningsoftwaredocumentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that large language models, guided by chain-of-thought prompting, in-context examples, and a structured privacy taxonomy, can extract privacy behaviors from ordinary software documents and turn them into short privacy user stories. It claims that popular models such as GPT-4o and Llama 3 do this with F1 scores exceeding 0.8; the detailed evaluation reports a best overall F1 of 0.766 for Llama-3.3-70b, with action extraction reaching 0.930. The authors also show that supervised fine-tuning can substantially improve smaller open models, raising Llama-3.2-3b by 0.454 F1 on held-out documents, while additional preference-based tuning on this small dataset reduces accuracy. A sympathetic reader would care because privacy requirements are often an afterthought in software projects, and automating their elicitation from documentation developers already produce could make privacy engineering more accessible.

What carries the argument

The argument rides on three connected objects. First, an extended version of the PAcT taxonomy supplies the fixed label vocabulary — 3 actions, 50 data types, and 26 purposes — that annotators used for the gold data and that the prompt instructs models to reuse. Second, the privacy story template 'We (action) (data type) for (purpose)' gives a minimal structured output for requirements. Third, the prompt template combines a task description with the taxonomy, one in-context example selected by sentence-embedding similarity to the input, XML-tagged output formatting for actions, data types, purposes, stories, and rationale, and a final instruction to restate the task and verify the answer. This chain-of-thought-style prompt is what converts raw documentation into structured extractions, and the two-stage tuning pipeline of supervised fine-tuning followed by preference optimization is what adapts smaller models to the task.

What would settle it

Run the same extraction and story-generation task on an independent corpus of software documents annotated by multiple external coders with reported inter-rater agreement, and have professional requirements engineers judge the generated stories; if the models' F1 falls well below the reported range (for example, below 0.5 for data types) or the stories are judged no better than template fills, the central claim would be falsified.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM prompted with a taxonomy of privacy labels and a simplified story template can reproduce the work of a human privacy analyst: given requirements documents, developer guides, architecture notes, or README files, the model identifies the actions, data types, and purposes that form privacy behaviors and expresses them as 'We (action) (data type) for (purpose)' user stories. On the authors' annotated corpus of 25 software document sets, the best base model, Llama-3.3-70b, achieves an overall F1 of 0.766 for behavior extraction, with 0.641 for data types and 0.762 for purposes, and the abstract summarizes these results as 'F1 scores exceeding 0.8.' Human annotators judged 73.3% of the stories generated by that model accurate, while also noting that about 61% of the manually written stories in the dataset were not produced by the model. The authors further claim that supervised fine-tuning brings a small model to within reach of the large model (Llama-3.1-8b reaches 0.766 F1 on held-out data), but that thought-preference optimization overfits this small dataset.

Load-bearing premise

The ground-truth dataset of 25 documents annotated by the authors themselves, without inter-annotator agreement metrics, is both correct and representative of real software documentation; if the labels are idiosyncratic or the documents unrepresentative, all F1 scores and story evaluations are artifacts of the authors' own annotation choices.

Editorial extensions

If this is right

  • Development teams could generate draft privacy requirements from documentation they already write, catching privacy behaviors earlier in the software lifecycle without a dedicated privacy expert.
  • Prompt design is a major lever: the final prompt outperformed the base prompt by 0.299–0.722 F1 across models, so structured formatting with an example and verification materially changes results.
  • Data types are the hardest category to extract, with best F1 of 0.641, and models tend to invent labels outside the taxonomy, so reliable data-type extraction will need constrained decoding or larger training sets.
  • Supervised fine-tuning can lift small open-source models to near the level of far larger models (a 0.454 F1 gain for Llama-3.2-3b), suggesting domain tuning can reduce dependence on proprietary APIs, but preference optimization overfits on small data.
  • The same prompting-and-tuning pipeline can likely be extended to other non-functional requirements, such as security and AI ethics, as the authors hypothesize.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The abstract's 'exceeding 0.8' figure is not supported by the best overall base-model F1 (0.766); the claim holds for action extraction and for some fine-tuned held-out results, so readers should treat 0.766 as the representative overall number.
  • The 61% recall of ground-truth stories in the human evaluation implies that even an accurate story set may omit many privacy behaviors, which matters more for compliance than for drafting support.
  • Because the corpus was assembled partly from LLM-suggested links and apps already documenting data practices, teams with sparse documentation may see lower performance; retrieval-augmented generation, which the authors flag as future work, would be the natural remedy.
  • A controlled ablation separating XML output formatting from the verification step would pinpoint which prompt component carries the large F1 gains the paper attributes to the final template.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper proposes a framework that uses large language models (LLMs) with chain-of-thought prompting, in-context learning, supervised fine-tuning, and direct preference optimization to extract privacy behaviors (actions, data types, purposes) from software documentation and generate privacy user stories. The authors created a manually annotated dataset of 25 software document sets from 20 open-source applications, extended the PAcT privacy taxonomy, and evaluated several commercial and open-source LLMs. They report F1 scores for behavior extraction, human evaluation of generated stories, and improvements from parameter tuning, concluding that LLMs can partially automate privacy requirements elicitation.

Significance. If the findings hold, the paper would contribute a useful tool for early-stage privacy requirements engineering and a new annotated dataset for evaluating LLMs on privacy-related software documentation. The use of chain-of-thought and in-context learning in this domain is a reasonable extension of prior work. However, the evaluation's validity is currently undermined by several internal inconsistencies and by the lack of an independent test set, which limits the significance of the reported numeric results until those issues are addressed.

major comments (5)
  1. [Abstract; Section I (RQ1); Section V-A] The abstract and the introduction's RQ1 claim that models such as GPT-4o and Llama 3 achieve F1 scores exceeding 0.8 for extracting actions, data types, and purposes as privacy behaviors. This is contradicted by Table I, where the best overall F1 is 0.766 (Llama-3.3-70b), the best data-type F1 is 0.641, and the best purpose F1 is 0.762. Only the action subscore exceeds 0.8. The claim must be revised to match the reported numbers.
  2. [Section V-A] The parsing rule stated in Section V-A, "When parsing LLM responses, we exclusively gather labels that match the labels in our taxonomy," discards all model outputs that are not in the taxonomy before precision and recall are computed. In standard evaluation, such out-of-vocabulary predictions are false positives, so this filtering inflates the reported F1 scores. The paper itself notes in Section V-B that 26.7% of generated stories contain hallucinated behaviors not in PAcT; those hallucinations are exactly the outputs being filtered. The paper should report unfiltered metrics alongside the filtered ones, or explicitly count out-of-taxonomy labels as errors.
  3. [Section IV-A] The taxonomy inventory is internally inconsistent. Section IV-A says "the extended PAcT includes 3 actions, 50 data types, and 26 purpose labels," but the same paragraph says the ground-truth dataset consists of "171 privacy behaviors (i.e., 50 actions, 60 data types, and 61 purposes)." The action count changes from 3 to 50, and the data-type and purpose counts also differ. The paper needs to clarify whether some counts refer to taxonomy nodes and others to annotated instances, and to document the final label inventory precisely for reproducibility.
  4. [Section IV-A; Section V] The evaluation is self-referential: the taxonomy and gold labels were created by the authors on the same 25 documents used for evaluation, and Section IV-A states the annotators "did not calculate the Kappa agreement metrics." There is no independent held-out annotation or external benchmark, so the reported F1 scores reflect in-sample agreement between the models and the authors' own labeling choices. The paper should either conduct an independent annotation study with agreement measures or explicitly present the results as a feasibility demonstration on a small, in-house dataset rather than as generalizable accuracy.
  5. [Section V (Tables I, II, III, V); Section VI] No confidence intervals, standard deviations, or statistical significance tests are reported for any model comparison, prompt comparison, or SFT/DPO improvement. With only 25 documents and a single run per configuration, the numerical differences—such as the prompt improvement range 0.299–0.722 in Section V-A or the SFT gains in Section V-C—may be within noise. The paper should report multiple runs, bootstrapped confidence intervals, or hypothesis tests to support its comparative claims.
minor comments (6)
  1. [Section VI-B] The limitation text says "only privacy stories generated by Llama3.2-70b were manually reviewed," but the model evaluated elsewhere is Llama-3.3-70b. This appears to be a typo and should be corrected.
  2. [Throughout] The naming of the GPT model is inconsistent: the abstract and text use "GPT-4o," while tables and some sections use "GPT4o" or "GPT4.0." Please standardize the notation.
  3. [Table II] Table II reports a README F1 of 0.0 for Llama-3.2-3b, but the text does not discuss this degenerate case. A brief explanation would help the reader interpret the result.
  4. [Section IV-A] The annotation process is described with shifting roles ("first three authors," "second and third authors," "first author"), which makes it difficult to determine who labeled what and how conflicts were resolved. A clearer description of the annotation procedure would improve reproducibility.
  5. [References] Reference [44] appears to be an incomplete citation with no title or author list. Please verify and complete all references.
  6. [Section IV-B] The paper states that the default temperature value (0.7) was kept for all models, but it is not clear whether this applies to every evaluation and tuning run. Please state the temperature explicitly for each experimental condition.

Circularity Check

2 steps flagged · score 7.0 of 10

Central F1 scores are partly constructed: the ICL example can be the input document's own gold labels, and the parser discards out-of-taxonomy outputs before scoring.

  1. self definitional [Section IV-B, Prompt Template Design, second paragraph]
    "First, we provide an ICL example, which is a document from our ground-truth dataset with its privacy behaviors and stories. To find the best example document, we use sentence transformers [59] to generate embeddings for all 25 documents. We then compute cosine similarity over these embeddings to select the document that is most semantically aligned with the input document."

    By the paper's own procedure, the candidate ICL examples are the 25 ground-truth documents, and the selected example is the one 'most semantically aligned with the input document.' The input document is one of those 25; a document is maximally similar to itself, and no exclusion of the input document is stated. Because the ICL example includes that document's gold privacy behaviors and stories, the prompt can contain the exact target labels for the document being scored. The extraction/story-generation result is then read out of the prompt rather than independently produced, so the reported F1 'prediction' is self-definitional: the gold answer is an input by construction.

  2. other [Section V-A, Analysis of Privacy Behaviors' Extraction Task, first paragraph]
    "When parsing LLM responses, we exclusively gather labels that match the labels in our taxonomy."

    The F1 computation is restricted to model outputs that already belong to the extended PAcT label set. Any label invented by the model is dropped before precision and recall are calculated, although Section V-B counts hallucinated labels as a failure mode (26.7% of generated stories contain behaviors not in PAcT). Under standard evaluation, an out-of-vocabulary prediction is a false positive; discarding it removes exactly the model's known errors from the denominator. The reported F1 therefore measures performance on a filtered subset of outputs that match the authors' taxonomy, not raw extraction accuracy, making the headline result depend on the parser's construction.

full rationale

This paper is an empirical LLM evaluation rather than a formal derivation, so most of its logic is not circular in the mathematical sense. The author-created gold labels and the absence of inter-annotator agreement are validity threats, not circularity. However, two stated evaluation choices make the reported F1 scores partially self-constructed. First, the ICL example is selected as the most semantically aligned document among the same 25 ground-truth documents; as written, no exclusion of the input document is stated, so the prompt can contain the very gold behaviors and stories the model is asked to produce. Second, the parser discards all model outputs not in the taxonomy before scoring, so hallucinated labels that the paper itself identifies as a failure mode never count as false positives. Both choices inflate the headline 'F1 exceeding 0.8' claim, which is also not supported by Table I (maximum overall F1 0.766); that inconsistency is a correctness issue rather than circularity. Some held-out parameter-tuning results in Table V do exceed 0.8 and provide partial independent content, so the paper is not wholly vacuous. Overall score 7 reflects that the central empirical claim is substantially circular by construction while retaining some independent held-out and qualitative evidence.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claim depends on a small, author-created dataset and label taxonomy, with no independent benchmark. The free parameters are hand-chosen labels, a fixed temperature, and a resource-constrained fine-tuning configuration. The main assumptions concern the validity and representativeness of the gold labels and the suitability of the story template.

free parameters (3)
  • Extended taxonomy labels (3 actions, 50 data types, 26 purposes) = 79 added labels
    Created by open coding the same 25 documents until saturation; these labels define the ground truth that all F1 scores are measured against.
  • Prompt temperature = 0.7
    Kept at the model default deliberately to increase output diversity; not tuned or varied in the experiments.
  • SFT and DPO training configuration = batch 2, grad accum 3, lr 5e-6, 3 epochs, LoRA rank 64, beta 0.1
    Chosen to fit a free Google Colab T4 environment rather than via hyperparameter search; the tuned model results depend on these choices.
assumptions (5)
  • domain assumption The authors' manual annotations are a valid gold standard.
    Section IV-A: no Kappa agreement was calculated, annotators are authors, and the same labels serve as ground truth for all evaluations.
  • domain assumption The 25-document corpus represents SDLC documents for privacy-relevant apps.
    Section IV-A and Section VI-B: the sample is a convenience sample from GitHub, F-Droid, and LLM suggestions, and the authors acknowledge limited transferability.
  • domain assumption The privacy story template captures privacy requirements adequately.
    Section II-B: the template is defined by the authors without external validation or comparison to other requirements formats.
  • standard math Hierarchical F1 scoring is the right evaluation metric.
    Section V-A: adopted from Kiritchenko et al. for hierarchical labels; no alternative metric is discussed.
  • ad hoc to paper Cosine-similarity-based ICL example selection improves accuracy.
    Section IV-B: the example is selected from the same 25 documents, with no ablation comparing selection strategies.
invented entities (1)
  • Privacy story template, 'The Application (action) (data type) for (purpose)'
    purpose: Converts extracted privacy behaviors into user-story style privacy requirements.
    No external validation that this template is an adequate requirements artifact; it rests on the authors' simplification of user story conventions (Section II-B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Generating Privacy Stories From Software Documentation." pith.science (2026). https://pith.science/paper/4O4I52MM

@misc{pith2026250623014,
  author       = {Pith},
  title        = {Pith review of: Generating Privacy Stories From Software Documentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4O4I52MM}},
  note         = {Machine review of arXiv:2506.23014}
}
read the original abstract

Research shows that analysts and developers consider privacy as a security concept or as an afterthought, which may lead to non-compliance and violation of users' privacy. Most current approaches, however, focus on extracting legal requirements from the regulations and evaluating the compliance of software and processes with them. In this paper, we develop a novel approach based on chain-of-thought prompting (CoT), in-context-learning (ICL), and Large Language Models (LLMs) to extract privacy behaviors from various software documents prior to and during software development, and then generate privacy requirements in the format of user stories. Our results show that most commonly used LLMs, such as GPT-4o and Llama 3, can identify privacy behaviors and generate privacy user stories with F1 scores exceeding 0.8. We also show that the performance of these models could be improved through parameter-tuning. Our findings provide insight into using and optimizing LLMs for generating privacy requirements given software documents created prior to or throughout the software development lifecycle.

Figures

Figures reproduced from arXiv: 2506.23014 by the authors.

Figure 1
Figure 1. An Example of a Privacy Story ing actions, data types, and purposes as privacy behaviors. Similarly, these models generate privacy stories with more than 80% precision. According to our annotators, only 16% of these accurate stories could be improved. - RQ2: What are the LLMs’ challenges to accurately generate privacy stories given software documents? In our quantitative and qualitative analysis, we found that using… view at source ↗
Figure 2
Figure 2. An Excerpt of Our Extended Privacy Action Taxonomy [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of Our Framework for Generating Privacy Stories [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: LLM Annotation Prompt Template output). To accomplish this, we follow insights from prior research [31], [60] that show that using XML tags within prompts may improve the structure for formatting model inputs and outputs. Thus, our prompt templates instruct the model t…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 52 canonical work pages

  1. [1]

    Privacy engineering: Shaping an emerging field of research and practice,

    S. G ¨urses and J. M. del Alamo, “Privacy engineering: Shaping an emerging field of research and practice,” IEEE Security & Privacy , vol. 14, no. 2, pp. 40–46, Mar 2016

    work page 2016

  2. [2]

    Rethinking the proposition of privacy engi- neering,

    A. Ceross and A. Simpson, “Rethinking the proposition of privacy engi- neering,” 2024, accessed: October 10, 2024. [Online]. Available: https: //ora.ox.ac.uk/objects/uuid:f59a9106-155c-43c2-b3da-dc986240ad03

    work page 2024

  3. [3]

    Privacy by designers: Software developers’ privacy mindset,

    I. Hadar and et al., “Privacy by designers: Software developers’ privacy mindset,” Empirical Software Engineering, vol. 23, no. 1, pp. 259–289, Apr 2017

    work page 2017

  4. [4]

    Evaluating privacy perceptions, experience, and behavior of software develop- ment teams,

    M. Prybylo, S. Haghighi, S. T. Peddinti, and S. Ghanavati, “Evaluating privacy perceptions, experience, and behavior of software develop- ment teams,” in Twentieth Symposium on Usable Privacy and Security (SOUPS 2024), 2024, pp. 101–120

    work page 2024

  5. [5]

    Pact: Detecting and classifying privacy behavior of android applications,

    V . Jain and et al., “Pact: Detecting and classifying privacy behavior of android applications,” in WiSec ’22: 15th ACM Conf. on Security and Privacy in Wireless and Mobile Networks . ACM, 2022, pp. 104–118

    work page 2022

  6. [6]

    Towards fine- grained localization of privacy behaviors,

    V . Jain, S. Ghanavati, S. T. Peddinti, and C. McMillan, “Towards fine- grained localization of privacy behaviors,” in 2023 IEEE 8th European Symp. on Security and Privacy (EuroS&P) , 2023, pp. 258–277

    work page 2023

  7. [7]

    A privacy threat analysis framework: supporting the elicitation and fulfillment of privacy requirements,

    M. Deng, K. Wuyts, R. Scandariato, B. Preneel, and W. Joosen, “A privacy threat analysis framework: supporting the elicitation and fulfillment of privacy requirements,” Requirements Engineering, vol. 16, pp. 3–32, 2011, published: 16 November 2010. [Online]. Available: https://doi.org/10.1007/s00766-010-0115-7

    doi:10.1007/s00766-010-0115-7 2011

  8. [8]

    Privacy requirements elicitation: a systematic literature review and perception analysis of it practitioners,

    E. D. Canedo, I. N. Bandeira, A. T. Calazans, and C. Cappelli, “Privacy requirements elicitation: a systematic literature review and perception analysis of it practitioners,” Requirements Engineering , vol. 28, pp. 177–194, 2023, published: 11 June 2022. [Online]. Available: https://doi.org/10.1007/s00766-022-00382-8

    work page doi:10.1007/s00766-022-00382-8 2023

Show all 73 references
  1. [9]

    Legal goal-oriented require- ment language (legal grl) for modeling regulations,

    S. Ghanavati, D. Amyot, and A. Rifaut, “Legal goal-oriented require- ment language (legal grl) for modeling regulations,” in Proc. of the 6th Int. Workshop on modeling in software engineering , 2014, pp. 1–6

    2014

  2. [10]

    Addressing privacy requirements in system design: the pris method,

    C. Kalloniatis, E. Kavakli, and S. Gritzalis, “Addressing privacy requirements in system design: the pris method,” Requirements Engineering, vol. 13, pp. 241–255, 2008, published: 07 August 2008. [Online]. Available: https://doi.org/10.1007/s00766-008-0067-3

    2008 doi

  3. [11]

    Privacystory: tool support for extracting privacy requirements from user stories,

    G. B. Herwanto, G. Quirchmayr, and A. M. Tjoa, “Privacystory: tool support for extracting privacy requirements from user stories,” in 2022 IEEE 30th Int. Requirements Engineering Conf. , 2022, pp. 264–265

    2022

  4. [12]

    Detecting privacy require- ments from user stories with nlp transfer learning models,

    F. Casillo, V . Deufemia, and C. Gravino, “Detecting privacy require- ments from user stories with nlp transfer learning models,” Information and Software Technology, vol. 146, 2022, art. no. 106853

    2022

  5. [13]

    Leveraging NLP techniques for privacy requirements engineering in user stories,

    G. B. Herwanto, G. Quirchmayr, and A. M. Tjoa, “Leveraging NLP techniques for privacy requirements engineering in user stories,” IEEE Access, 2024

    2024

  6. [14]

    Gdpr-based user stories in the access control perspective,

    C. Bartolini, S. Daoudagh, G. Lenzini, and E. Marchetti, “Gdpr-based user stories in the access control perspective,” inInt. Conf. on the Quality of Information and Communications Tech. Springer, 2019, pp. 3–17

    2019

  7. [15]

    Modeling security and privacy requirements: a use case- driven approach,

    P. X. Mai, A. Goknil, L. K. Shar, F. Pastore, L. C. Briand, and S. Shaame, “Modeling security and privacy requirements: a use case- driven approach,” Information and Software Technology , vol. 100, pp. 165–182, 2018

    2018

  8. [16]

    Investigating chatgpt’s potential to assist in requirements elicitation processes,

    K. Ronanki, C. Berger, and J. Horkoff, “Investigating chatgpt’s potential to assist in requirements elicitation processes,” in 49th Euromicro Conf. on SE and Advanced Applications . IEEE, 2023, pp. 354–361

    2023

  9. [17]

    Using llms in software requirements specifications: An empirical evaluation,

    M. Krishna, B. Gaur, A. Verma, and P. Jalote, “Using llms in software requirements specifications: An empirical evaluation,” in 2024 IEEE 32nd Int. Requirements Engineering Conf. , 2024

    2024

  10. [18]

    Language models are few-shot learners,

    Brown and et al., “Language models are few-shot learners,” Advances in Neural Information Processing Systems (NeurIPS) , vol. 33, 2020

    2020

  11. [19]

    Chain-of-thought prompting elicits reasoning in large language models chain-of-thought prompting,

    J. Wei and et al., “Chain-of-thought prompting elicits reasoning in large language models chain-of-thought prompting,” 2022. [Online]. Available: https://arxiv.org/pdf/2201.11903

    2022 arXiv

  12. [20]

    Xiao and J

    T. Xiao and J. Zhu, Foundations of Large Language Models . arXiv, 2025, arXiv:2501.09223. [Online]. Available: https://doi.org/10.48550/ arXiv.2501.09223

    2025 doi arXiv

  13. [21]

    Large language models for data annotation and synthesis: A survey,

    Z. Tan, D. Li, S. Wang, A. Beigi, B. Jiang, A. Bhattacharjee, M. Karami, J. Li, L. Cheng, and H. Liu, “Large language models for data annotation and synthesis: A survey,” arXiv, vol. 2402.13446, 2024, arXiv:2402.13446 [cs.CL], Accepted to EMNLP 2024 Main. [Online]. Available: ...

    2024 doi arXiv

  14. [22]

    Requirements satisfiability with in-context learning,

    S. Santos, T. Breaux, T. Norton, S. Haghighi, and S. Ghanavati, “Requirements satisfiability with in-context learning,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE) . IEEE, 2024, pp. 168–179

    2024

  15. [23]

    Large language models: A new approach for privacy policy analysis at scale,

    D. Rodriguez, I. Yang, J. M. Del, and N. Sadeh, “Large language models: A new approach for privacy policy analysis at scale,” Computing, Aug 2024

    2024

  16. [24]

    Toward regulatory compliance: A few-shot learning approach to extract processing activities,

    K. Pragyan, R. Ghandiparsi, R. Slavin, S. Ghanavati, T. Breaux, and M. B. Hosseini, “Toward regulatory compliance: A few-shot learning approach to extract processing activities,” in 2024 IEEE 32nd Interna- tional Requirements Engineering Conference Workshops (REW). IEEE, 2024,...

    2024

  17. [25]

    Ppchecker: Towards accessing the trustworthiness of android apps’ privacy policies,

    L. Yu, X. Luo, J. Chen, H. Zhou, T. Zhang, H. Chang, and H. K. Leung, “Ppchecker: Towards accessing the trustworthiness of android apps’ privacy policies,” IEEE Transactions on Software Engineering , vol. 47, no. 2, pp. 221–242, 2018

    2018

  18. [26]

    On the ridiculousness of notice and consent: Contradictions in app privacy policies,

    E. Okoyomon, N. Samarin, P. Wijesekera, A. Elazari Bar On, N. Vallina- Rodriguez, I. Reyes, ´A. Feal, S. Egelman et al., “On the ridiculousness of notice and consent: Contradictions in app privacy policies,”In: Workshop on Technology and Consumer Protection (ConPro 2019), in c...

    2019

  19. [27]

    Maps: Scaling privacy compliance analysis to a million apps,

    S. Zimmeck, P. Story, D. Smullen, A. Ravichander, Z. Wang, J. Reiden- berg, N. C. Russell, and N. Sadeh, “Maps: Scaling privacy compliance analysis to a million apps,” In: Proceedings on Privacy Enhancing Technologies, pp. 66–86, 2019

    2019

  20. [28]

    Under- standing challenges for developers to create accurate privacy nutrition labels,

    T. Li, K. Reiman, Y . Agarwal, L. F. Cranor, and J. I. Hong, “Under- standing challenges for developers to create accurate privacy nutrition labels,” In: Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems , pp. 1–24, 2022

    2022

  21. [29]

    Unpacking privacy labels: A measurement and developer perspective on google’s data safety section,

    R. Khandelwal, A. Nayak, P. Chung, and K. Fawaz, “Unpacking privacy labels: A measurement and developer perspective on google’s data safety section,” in 33rd USENIX Security Symposium (USENIX Security 24) . Philadelphia, PA: USENIX Association, Aug. 2024, pp. 2831–2848. [Onlin...

    2024

  22. [30]

    Atlas: Auto- matically detecting discrepancies between privacy policies and privacy labels,

    A. Jain, D. Rodriguez, J. M. Del Alamo, and N. Sadeh, “Atlas: Auto- matically detecting discrepancies between privacy policies and privacy labels,” In: 2023 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW), pp. 94–107, 2023

    2023

  23. [31]

    Thinking llms: General instruction following with thought generation,

    T. Wu, J. Lan, W. Yuan, J. Jiao, J. Weston, and S. Sukhbaatar, “Thinking llms: General instruction following with thought generation,” Oct 2024

    2024

  24. [32]

    Data safety labels for developers,

    Google, “Data safety labels for developers,” https://support.google. com/googleplay/android-developer/answer/10787469?hl=en#zippy= %2Cdata-types, accessed: [Access Date]

  25. [33]

    Cohn, User stories applied: For agile software development

    M. Cohn, User stories applied: For agile software development . Addison-Wesley Professional, 2004

    2004

  26. [34]

    Beyond chain-of-thought: A survey of chain-of-x paradigms for llms,

    Y . Xia and et al., “Beyond chain-of-thought: A survey of chain-of-x paradigms for llms,” 2024, accessed: Oct. 10, 2024. [Online]. Available: https://arxiv.org/abs/2404.15676

    2024 arXiv

  27. [35]

    Reasoning with large language models, a survey,

    A. Plaat and et al., “Reasoning with large language models, a survey,”

  28. [36]

    A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains,

    A. Jacovi and et al., “A chain-of-thought is as strong as its weakest link: A benchmark for verifiers of reasoning chains,” 2024, accessed: Oct. 10, 2024. [Online]. Available: https://arxiv.org/abs/2402.00559

    2024 arXiv

  29. [37]

    Chain-of-verification reduces hallucination in large language models,

    S. Dhuliawala and et al., “Chain-of-verification reduces hallucination in large language models,” 2023. [Online]. Available: https://arxiv.org/ abs/2309.11495

    2023 arXiv

  30. [38]

    Haloscope: Harnessing unlabeled llm generations for hallucination detection,

    X. Du, C. Xiao, and Y . Li, “Haloscope: Harnessing unlabeled llm generations for hallucination detection,” 2024, accessed: Oct. 10, 2024. [Online]. Available: https://arxiv.org/abs/2409.17504

    2024 arXiv

  31. [39]

    Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models,

    F. Wang, X. Wan, R. Sun, J. Chen, and S. O. Arık, “Astute rag: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models,” 2024, accessed Oct. 10, 2024. [Online]. Available: https://arxiv.org/abs/2410.07176

    2024 arXiv

  32. [40]

    Fine-tuning language models from human preferences,

    D. M. Ziegler and et al., “Fine-tuning language models from human preferences,” arXiv:1909.08593 [cs, stat] , 2020. [Online]. Available: https://arxiv.org/abs/1909.08593

    1909 arXiv

  33. [41]

    Igniting language intelligence: The hitchhiker’s guide from chain-of-thought reasoning to language agents,

    Z. Zhang and et al., “Igniting language intelligence: The hitchhiker’s guide from chain-of-thought reasoning to language agents,” 2023. [Online]. Available: https://arxiv.org/abs/2311.11797

    2023 arXiv

  34. [42]

    Direct preference optimization: Your language model is secretly a reward model,

    R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn, “Direct preference optimization: Your language model is secretly a reward model,” in Proceedings of the 37th Conference on Neural Information Processing Systems (NeurIPS 2023) , 2023, arXiv:2305.18290v3...

    2023 arXiv

  35. [43]

    Leveraging llms for the quality assurance of software requirements,

    L. S. and et al., “Leveraging llms for the quality assurance of software requirements,” accessed Oct. 10, 2024. [Online]. Available: https://ieeexplore.ieee.org/document/10628462/

    2024

  36. [44]

    10, 2024

    Accessed Oct. 10, 2024. [Online]. Available: www.ijetajournal.org/ volume-3/issue-1/IJETA-V3I1P1.pdf

    2024

  37. [45]

    When llm-based code generation meets the software development process,

    F. Lin and et al., “When llm-based code generation meets the software development process,” Mar 2024. [Online]. Available: https://arxiv.org/abs/2403.15852

    2024 arXiv

  38. [46]

    Advancing requirements engineering through generative ai: Assessing the role of llms,

    C. Arora and et al., “Advancing requirements engineering through generative ai: Assessing the role of llms,” in Generative AI for Effective Software Development. SpringerLink, Jan 2024

    2024

  39. [47]

    Llm-based agents for automating the enhancement of user story quality: An early report,

    Z. Zhang and et al., “Llm-based agents for automating the enhancement of user story quality: An early report,” in International Conference on Agile Software Development . Springer, 2024

    2024

  40. [48]

    Extracting software functional re- quirements from free text documents,

    Y . Mu, Y . Wang, and J. Guo, “Extracting software functional re- quirements from free text documents,” in Proceedings of the 2009 International Conference on Information and Multimedia Technology . IEEE, 2009, pp. 194–198

    2009

  41. [49]

    Automated extraction of non-functional requirements in available documentation,

    J. Slankas and L. Williams, “Automated extraction of non-functional requirements in available documentation,” in 2013 1st International Workshop on Natural Language Analysis in Software Engineering (Nat- uraLiSE), San Francisco, CA, USA, 2013, pp. 9–16

    2013

  42. [50]

    Pure: A dataset of public re- quirements documents,

    A. Ferrari, G. O. Spagnolo, and S. Gnesi, “Pure: A dataset of public re- quirements documents,” in 2017 IEEE 25th International Requirements Engineering Conference (RE) , 2017, pp. 502–505

    2017

  43. [51]

    Extracting software requirements from unstructured documents,

    V . Ivanov, E. Pavlyshenko, Y . Ivanov, and A. Kolykhalova, “Extracting software requirements from unstructured documents,” 2022

    2022

  44. [52]

    Human-llm collaborative annotation through effective verification of llm labels,

    X. Wang, H. Kim, S. Rahman, K. Mitra, and Z. Miao, “Human-llm collaborative annotation through effective verification of llm labels,” May 2024

    2024

  45. [53]

    A comparative study on annotation quality of crowdsourcing and llm via label aggregation,

    J. Li, “A comparative study on annotation quality of crowdsourcing and llm via label aggregation,” arXiv (Cornell University) , Mar 2024

    2024

  46. [54]

    A large language model approach to code and privacy policy alignment,

    G. Morales, P. K. C., S. Jahan, M. B. Hosseini, and R. Slavin, “A large language model approach to code and privacy policy alignment,” Mar 2024

    2024

  47. [55]

    The creation and analysis of a website privacy policy corpus,

    S. Wilson, F. Schaub, A. A. Dara, F. Liu, S. Cherivirala, P. G. Leon, M. S. Andersen, S. Zimmeck, K. M. Sathyendra, N. C. Russell et al. , “The creation and analysis of a website privacy policy corpus,” in Proceedings of the 54th Annual Meeting of the Association for Com- puta...

    2016

  48. [56]

    Policygpt: Automated analysis of privacy policies with large language models,

    C. Tang, D. Lin, Y . Li, and S. Sun, “Policygpt: Automated analysis of privacy policies with large language models,” 2023

    2023

  49. [57]

    F-droid: Free and open source android app repository,

    F-Droid Community, “F-droid: Free and open source android app repository,” accessed: 2024-03-07. [Online]. Available: https: //f-droid.org

    2024

  50. [58]

    Conceptualization: On theory and theorizing using grounded theory,

    B. G. Glaser, “Conceptualization: On theory and theorizing using grounded theory,” International journal of qualitative methods , vol. 1, no. 2, pp. 23–38, 2002

    2002

  51. [59]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” 2019

    2019

  52. [60]

    Hot: Highlighted chain of thought for referencing supporting facts from inputs,

    T. Nguyen, L. Bolton, M. R. Taesiri, and A. T. Nguyen, “Hot: Highlighted chain of thought for referencing supporting facts from inputs,” arXiv, vol. 2503.02003, 2025, arXiv:2503.02003 [cs.CL]. [Online]. Available: https://doi.org/10.48550/arXiv.2503.02003

    2025 doi arXiv

  53. [61]

    A survey of large language models,

    W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y . Hou, Y . Min, B. Zhang, J. Zhang, Z. Dong, Y . Du, C. Yang, Y . Chen, Z. Chen, J. Jiang, R. Ren, Y . Li, X. Tang, Z. Liu, P. Liu, J.-Y . Nie, and J.-R. Wen, “A survey of large language models,” arXiv, vol. 2303.18223, 2023, arX...

    2023 doi arXiv

  54. [62]

    Unsloth: Efficient fine-tuning for llms,

    U. Team, “Unsloth: Efficient fine-tuning for llms,” accessed Dec. 9,

  55. [63]

    Transformers reinforcement learning (trl),

    H. Face, “Transformers reinforcement learning (trl),” accessed Dec. 9,

  56. [64]

    Available: github.com/unslothai/unsloth

    [Online]. Available: github.com/unslothai/unsloth

  57. [65]

    The llama 3 herd of models,

    A. Dubey, Y . Li, P. Dabas, L. Zeng, C. C. Wu, and A. Cattelan, “The llama 3 herd of models,” July 2024

    2024

  58. [66]

    Available: github.com/huggingface/trl

    [Online]. Available: github.com/huggingface/trl

  59. [67]

    Gpt-4 technical report,

    OpenAI, G. Brockman, M. Brown, R. Cheever, A. H. Chung, T. Jun, D. Krueger, A. Law, L. Leike, and K. B. Miller, “Gpt-4 technical report,” Mar 2023

    2023

  60. [68]

    Software systems documentation: A systematic review,

    A. A. H. Alzahrani, “Software systems documentation: A systematic review,” International Journal of Advanced Computer Science and Applications, vol. 15, no. 8, pp. 155–162, 2024

    2024

  61. [69]

    Qwen2.5 technical report,

    A. Y . et al., “Qwen2.5 technical report,” 2025, arXiv preprint arXiv:2412.15115v2

    2025 arXiv

  62. [70]

    Learning and evaluation in the presence of class hierarchies: Application to text categorization,

    S. Kiritchenko, S. Matwin, R. Nock, and A. F. Famili, “Learning and evaluation in the presence of class hierarchies: Application to text categorization,” in Advances in Artificial Intelligence. Canadian AI 2006, L. Lamontagne and M. Marchand, Eds., vol. 4013. Springer, 2006

    2006

  63. [72]

    Agents companion,

    A. Gulli, L. Nigam, J. Wiesinger, V . Vuskovic, I. Sigler, I. Nardini, N. Stroppa, S. Kartakis, N. Saribekyan, A. Nawalgaria, and A. Bount, “Agents companion,” Kaggle / Google, Tech. Rep., February 2025, white Paper. [Online]. Available: https://www.kaggle. com/whitepaper-agen...

    2025

  64. [73]

    Building effective agents,

    Anthropic, “Building effective agents,” https://www.anthropic.com/ engineering/building-effective-agents, December 2024, blog post on Anthropic’s Engineering website

    2024

  65. [2024]

    Available: https://arxiv.org/html/2407.11511v1

    [Online]. Available: https://arxiv.org/html/2407.11511v1

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.