Pith. sign in

REVIEW 5 major objections 6 minor 68 references

This paper argues that off-the-shelf LLMs can replicate the functionalities of specialized privacy-policy analysis tools, matching or exceeding their performance across most tasks without fine-tuning.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:05 UTC pith:5QTYIJG4

load-bearing objection A genuinely useful first benchmark of specialized privacy-policy tools vs off-the-shelf LLMs, but the missing contamination check keeps the headline capability claim provisional. the 5 major comments →

arxiv 2607.17075 v1 pith:5QTYIJG4 submitted 2026-07-19 cs.CR cs.CL

A Systematic Evaluation of Traditional Privacy Policy Analysis Tools Against LLMs

Madhav Aryal , Sudipa Saha , Kaushal Kafle , Anshuman Chhabra , Sunil Manandhar This is my paper
classification cs.CR cs.CL
keywords large language modelsprivacy policy analysiscontradiction detectionregulatory compliancesemantic role labelingOPP-115policy annotationLLM evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper attempts to establish that off-the-shelf large language models, given only carefully worded prompts and no domain-specific fine-tuning, can reproduce the full range of functionality that specialized privacy-policy analysis tools provide. Across six tools spanning contradiction detection, regulatory compliance analysis, and policy aggregation, plus two intermediate tasks (semantic role labeling and manual annotation), the authors report that at least one LLM configuration matches or exceeds the tool on nearly every metric. For example, the best LLM configuration detects valid intra-policy contradictions that the tools miss entirely, and achieves higher F1 scores on GDPR completeness checks than the dedicated tools. On labeling first-party and third-party data practices, the LLMs' outputs approach the agreement rates of expert human annotators. If correct, this would mean that a single general-purpose model with good prompts could replace a patchwork of brittle, specialized tools, lowering the barrier to privacy research and changing how future tools are built.

Core claim

The central discovery is that prompting a state-of-the-art LLM with the same output format and definitions used by a specialized tool is sufficient to match or beat that tool's measured performance. In the study's head-to-head comparisons on ten real-world privacy policies, GPT-5.2 and Gemini-2.5-Pro outperformed PolicyLint and Poligraph in contradiction detection (the tools found zero validated contradictions, while the best LLM configuration found several at 26.3% precision), outperformed AutoCompliance and PolicyChecker in GDPR completeness checking (F1 of up to 94% vs the tools' 48% and 68.7%), and outperformed Poligraph in policy aggregation (F1 of 88.2% vs 67.1%). The LLMs also matched

What carries the argument

The evaluation harness is the key device: a set of tool-specific prompts that instruct an LLM to produce output in the exact format of each specialized tool, combined with a shared dataset of ten policies and manual validation to score results. The dichotomy of 'simple prompt' (no domain context) versus 'detailed prompt' (with definitions and rules from the tool's paper) functions as the independent variable, isolating how much of the LLM's success depends on domain knowledge supplied by the user. This prompt-based replication is what allows a direct, apples-to-apples comparison of an off-the-shelf model against a purpose-built system.

Load-bearing premise

The evaluation assumes that the LLMs have not memorized the benchmark texts or labels, so their prompt-driven performance reflects general capability rather than recall of the specific OPP-115 annotations or the ten popular privacy policies used as test data.

What would settle it

Run the same prompts on a set of privacy policies written after the LLMs' training cutoffs, with annotations produced by human experts and never published online; if the LLMs' precision and recall drop materially below their scores on OPP-115 and the ten Google Play policies, then memorization rather than analytic ability would explain the reported parity.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • A single off-the-shelf LLM with tailored prompts can replace multiple specialized tools for the studied tasks, so future research can allocate effort away from building separate classifiers for each privacy-analysis function.
  • LLM-based contradiction detection finds valid contradictions that rule- or pattern-based tools miss entirely, suggesting real-world privacy policies contain more internal inconsistencies than previously measured by automated tools.
  • Detailed prompts with definitions and domain context substantially improve precision on granular tasks like PolicyChecker and contradiction detection, though even simple prompts can beat several specialized tools.
  • LLM annotation of third-party sharing entities is comparable to expert human annotators (best F1 84.5% vs 76% Cohen's kappa), while first-party collection lags, indicating that semi-automated labeling pipelines are feasible for some categories.
  • Hybrid approaches that combine traditional tools (for high precision, low cost, and auditability) with LLMs (for recall, semantic reasoning, and flexibility) may be a more practical path than wholesale replacement.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper does not test for pretraining contamination - if the models memorized OPP-115 or the ten selected policies, the headline parity numbers could overstate general ability; a direct test would run the same prompts on policies written after the models' training cutoff with fresh human labels.
  • Editorial extension: because the paper observes substantial variance across prompts and models, ensembling multiple prompt strategies (or multiple models) could push recall and coverage beyond any single configuration, at increased API cost.
  • Editorial extension: the same prompt-based replication methodology could be applied to newer regional regulations (e.g., CPRA, PIPL, TDPSA) without retraining dedicated tools, since LLMs are not tied to the rulesets of the six evaluated tools.
  • Editorial extension: the finding that simple prompts already beat several specialized tools implies that current privacy-policy tools are brittle enough that general language understanding outperforms keyword and pattern rules; a natural next step is to test the same approach on a larger, more diverse corpus of policies.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents a systematic comparison of two state-of-the-art LLMs (GPT-5.2 and Gemini-2.5-Pro) against six specialized privacy-policy analysis tools, across three end-to-end functionalities (intra-policy contradiction detection, regulatory compliance analysis, and policy aggregation) and two intermediate tasks (semantic role labeling and manual annotation). Using a custom dataset of ten popular Google Play privacy policies and the public OPP-115 corpus, the authors prompt the LLMs with two strategies (a simple Prompt 1 and a detailed Prompt 2 that includes tool-specific definitions and output formats) and compare against the tools' outputs. They report that LLMs consistently match or exceed the tools, e.g., F1 of 94% vs 48% for AutoCompliance, 89.9% vs 68.7% for PolicyChecker, and 88.2% vs 67.1% for Poligraph aggregation. They also find that LLM annotations of OPP-115 categories approach inter-human agreement, and conclude that off-the-shelf LLMs can largely replace specialized tools, while advocating a hybrid approach that retains the scalability of traditional tools.

Significance. If the results hold, this is a valuable contribution to privacy-policy analysis: it provides the first broad benchmark of LLM capabilities against a suite of pre-LLM tools, offers reusable prompting strategies, and identifies structural trade-offs (cost, auditability, scalability) that are useful for the community. The paper also exhibits strengths: it ships prompts in an online artifact, includes robustness checks (system prompts, one-shot examples), and adopts a conservative stance by counting borderline manual-validations as false positives. However, the headline claim—that off-the-shelf LLMs can replace specialized tools—rests on an untested assumption of no training-data contamination and on several subjective evaluation steps. These threats, if unaddressed, would materially weaken the causal interpretation of the results.

major comments (5)
  1. [Section 3.1, 5.1, 7] No contamination check is performed for the benchmark texts or labels. The ten Google Play policies are for the most popular apps worldwide (ChatGPT, TikTok, Instagram, WhatsApp, Temu, Shein), and OPP-115 has been public since 2016 and is widely cited in privacy-policy ML research. Both are highly likely to appear in the pretraining data of GPT-5.2 and Gemini-2.5-Pro. The grounding test in Section 7 verifies only that cited source sentences appear in the input policy; it cannot detect memorization of answers or labels. Since the headline comparisons in Tables 1, 4, and 5 could be inflated by memorization, the central claim that LLMs 'can effectively perform' these tasks is not yet established. Add a contamination check—e.g., evaluate on policies collected after the models' knowledge cutoff, or probe the models' ability to reproduce outputs/labels without the policy text provided.
  2. [Section 3.2, 4.2.1, 4.3] The manual ground-truth construction and validation process is vulnerable to bias. For AutoCompliance and PolicyChecker, ground truth is manually constructed by the authors; for Poligraph and contradiction detection, outputs from both tools and LLMs are manually validated. The paper reports overall agreement rates (90.28% and 89.07%) but does not state whether the annotators were blinded to whether an output came from a tool or an LLM, nor does it describe the annotation protocol in sufficient detail. This matters because the central claim that LLMs 'match or exceed' tools depends on these subjective judgments. Provide a detailed protocol, independent annotation, and inter-rater reliability for ground-truth construction, and perform validation with annotators blinded to system identity.
  3. [Section 5.3, 5.4] The Semantic Role Labeling comparison is precision-only and based on a 10% stratified sample (118 sentences) plus one full policy (TikTok). The paper reports no recall or F1 for SRL because no ground-truth set is constructed. The claim that LLMs 'matched or vastly outperformed' Purpliance or PolicyPulse is therefore restricted to precision on a small sample, and the 'vast' margins (e.g., 98.7% vs 83.8% for TikTok) may not reflect overall quality. Either construct a complete SRL ground truth for a subset of policies to enable recall/F1 measurement, or explicitly limit the conclusions to precision on the sampled sentences.
  4. [Section 3.2, 4] Prompt 2 injects the tools' own definitions, ontologies, violation categories, and output schemas (e.g., PolicyLint's contradiction criteria, Poligraph's CCPA ontology, PolicyChecker's mandatory/legal-basis requirements) into the LLM. This is necessary for matching output formats, but it creates an asymmetry: the tool had to encode this domain knowledge through manual engineering or training, while the LLM receives it for free in the prompt. The 'off-the-shelf' claim is thus not fully representative of what a non-expert user could achieve without first studying the tool papers and extracting these definitions. The paper should discuss this asymmetry and, ideally, evaluate a 'tool-agnostic expert prompt' that supplies only general privacy-policy regulatory knowledge, not the specific tool's rules.
  5. [Section 6.2, 7] The evaluation uses only ten policies, and the paper provides no statistical significance testing or confidence intervals. While some differences (e.g., AutoCompliance F1 48% vs 94%) are clearly large, others (e.g., PolicyChecker F1 89.9% vs 89.4% across configurations; Poligraph aggregation 88.2% vs 86.3%) may be within random variation. The paper acknowledges the small dataset but still draws general conclusions such as 'LLMs consistently match or exceed' and 'Prompt 2 significantly outperforms Prompt 1.' Add bootstrap confidence intervals or other uncertainty quantification where feasible, and temper claims that go beyond descriptive comparison on this dataset.
minor comments (6)
  1. [Section 4] Findings F8 and F9 are each used twice (in Section 4.2.1 for Poligraph compliance and in Section 4.3 for policy aggregation). Renumber to avoid ambiguity.
  2. [Table 1] The caption says 'Tasks marked with * lack ground truth and report precision (P) and agreement count (N) only.' This is stated in the text but not in the table caption; please move the explanation to the caption or footnote.
  3. [Section 7] The system-prompt robustness check is performed only for contradiction detection (PolicyLint), yet the conclusion 'results did not change substantially' is used to support the general study. Report analogous checks for at least one other task (e.g., AutoCompliance or aggregation), or temper the claim.
  4. [Section 3.1] 'To perform a generalizable, yet feasible comparison analysis' overstates generalizability for n=10. Recommend rephrasing to 'to enable a comparison across multiple categories within a feasible budget.'
  5. [Appendix] Table numbering is inconsistent in cross-references: Section 5.2 refers to 'Table 2 in Appendix' and 'Table 3 in Appendix,' but the tables are labeled Table 2 and Table 3 in the manuscript. Verify all cross-references.
  6. [Section 4.1] Typo: 'progvided' should be 'provided.' Also in Section 6.2, 'One one hand' should be 'On the one hand.'

Circularity Check

0 steps flagged

No circularity found: this is an empirical benchmark, not a derivation; Prompt 1 provides a definition-free baseline and no fitted quantity is relabeled as a prediction.

full rationale

This paper is an empirical evaluation rather than a derivation chain. The central comparison—LLMs vs six specialized tools on ten policies and OPP-115 annotations—does not fit parameters to target outputs and then claim those outputs as predictions. For each functionality, the authors use two prompting strategies: Prompt 1 with no domain definitions, and Prompt 2 that transparently supplies the tool's definitions, ontology, and output format. The existence of Prompt 1 is important: the headline results do not reduce by construction to the tool definitions, because the paper also reports that Prompt 1 alone outperforms several tools (e.g., Table 1: AutoCompliance F1 85% for Gemini-Prompt 1 vs 48% for the tool). The manually constructed ground truths for F_N2 and F_N3 are independent benchmarks derived from the policy corpus and regulatory requirements, not from the LLM outputs. Manual validation by the authors is a standard evaluation procedure and not a fitted input. The only self-citations ([13] artifact link and [34], a prior paper by co-authors) are not load-bearing for the central claim. Section 7's grounding test addresses whether LLM outputs trace to the input policy text; it does not test pretraining contamination, but contamination is a validity threat external to circularity. No step equates a prediction with an input by construction, and no uniqueness theorem or ansatz is imported from the authors' own prior work to force the result.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

This is an empirical benchmark paper, so there are no fitted parameters and no newly postulated entities. The central claims rest on domain assumptions about dataset representativeness, correctness of author-made ground truth, absence of LLM benchmark contamination, and faithful tool deployment.

axioms (4)
  • domain assumption The ten selected Google Play privacy policies are representative enough to support generalizable claims about LLMs replacing privacy tools.
    Section 3.1 selects ten policies from top free apps; Section 7 acknowledges findings 'may not generalize to other privacy policies.' The broad conclusion depends on this representativeness.
  • domain assumption Manual ground truths for contradiction detection, GDPR compliance, and aggregation are correct and unbiased.
    Sections 4.1-4.3: ground truth is constructed by the authors and outputs are manually validated by two authors; this is a plausible but not independently audited premise.
  • domain assumption The LLMs have not memorized OPP-115 labels or the ten popular privacy policies (no benchmark contamination).
    Section 5.1 evaluates against public OPP-115; Section 7 discusses dataset size but never tests for pretraining contamination, which could inflate every main comparison.
  • domain assumption The six selected tools were deployed faithfully on the local machine and their outputs represent their true capability.
    Section 3.1 restricts to tools deployable on a MacBook Air M4, 24GB RAM; a tool that fails to deploy or requires different hardware is excluded, which may bias the comparison.

pith-pipeline@v1.3.0-alltime-deepseek · 3350 in / 3629 out tokens · 147515 ms · 2026-08-01T19:05:20.512662+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of A Systematic Evaluation of Traditional Privacy Policy Analysis Tools Against LLMs." pith.science (2026). https://pith.science/paper/5QTYIJG4

@misc{pith2026260717075,
  author       = {Pith},
  title        = {Pith review of: A Systematic Evaluation of Traditional Privacy Policy Analysis Tools Against LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5QTYIJG4}},
  note         = {Machine review of arXiv:2607.17075}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

The advent of LLMs has significantly changed the research on privacy policy and data compliance analysis by enabling tasks that previously required specialized, domain-specific tools. However, it remains unclear to what extent LLMs can truly replicate the diverse functionalities, and the wide range of methodologies and analysis offered by prior work. In this paper, we conduct the first systematic evaluation of whether off-the-shelf LLMs can replace specialized privacy analysis tools. We study six representative tools spanning three major functionalities: contradiction detection, regulatory compliance analysis, and privacy policy summarization and aggregation, and across three intermediate tasks: structured data extraction using tuples, Semantic Role Labeling (SRL) and manual privacy policy labeling. We compare the performance of two state-of-the-art LLMs (GPT-5.2 and Gemini-2.5 in various configurations) against the tools by directly prompting the models to perform corresponding functionalities and tasks on a custom dataset of 10 privacy policies, allowing us to assess whether off-the-shelf models can produce tool-specific functionalities without further engineering or domain-specific training, major limitations in prior work. Our results show that LLMs consistently match or exceed the capabilities of existing tools across the functionalities. In manual labeling of first-party collection entities, LLMs achieved an average precision of 81.8% and recall of 70.9%, while for labeling of third-party sharing entities, they achieved an average precision of 91.4% and recall of 70.8% compared to the OPP-115 dataset. Overall, our findings indicate that LLMs can effectively perform a broad range of functionalities and tasks in privacy policy and regulation analysis that previously required specialized tools.

Figures

Figures reproduced from arXiv: 2607.17075 by Anshuman Chhabra, Kaushal Kafle, Madhav Aryal, Sudipa Saha, Sunil Manandhar.

Figure 1
Figure 1. Figure 1: Data Class Level Precision: Purpliance vs. LLMs. [PITH_FULL_IMAGE:figures/full_fig_p015_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Data Class Level Precision: PolicyPulse vs. LLMs. [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

68 extracted references · 2 linked inside Pith

  1. [1]

    https://www.oag.ca.gov/ privacy/ccpa

    California consumer privacy act (ccpa). https://www.oag.ca.gov/ privacy/ccpa. Accessed June 2026

    2026

  2. [2]

    https://leginfo.legislature.ca

    California privacy rights act (cpra). https://leginfo.legislature.ca. gov/faces/codes displayText.xhtml?division=3.&part=4.&lawCode= CIV&title=1.81.5. Accessed June 2026

    2026

  3. [3]

    https://www.flsenate.gov/Session/ Bill/2023/262/BillText/er/HTML

    Florida digital bill of rights (fdbr). https://www.flsenate.gov/Session/ Bill/2023/262/BillText/er/HTML. Accessed June 2026

    2023

  4. [4]

    we do not collect any personal informa- tion

    Functionality Replication with LLMs (DG 1) We selected three functionalities (labeled F NS ) for replication with LLMs– Intra-policy Contradiction Detec- tion, Regulatory Compliance Analysis, and Policy Aggre- gation – based on tool availability within our deployment criteria (Section 3.1), and compared the performance against the corresponding tools. As ...

  5. [5]

    We may share your information with

    Intermediate Task Replication with LLMs (DG 2) WhileDG 1 evaluated the ability of LLMs to replicate end-user functionalities of tools, these tools are reliant on in- termediate NLP tasks to process unstructured policy text and transform it into structured representations to execute the functionality-level analysis. Such tasks often require manual labor or...

  6. [6]

    per-query

    Trade-offs of LLMs vs Existing Tools (DG 3) While Sections 4 and 5 demonstrated to what extent off- the-shelf LLMs can successfully replicate both the end-user functionalities as well as the intermediate tasks performed by traditional tools (i.e.,RQ 1 andRQ 2), we discuss the trade-offs between both approaches in this section, given that they differ subst...

    1956

  7. [7]

    ungrounded

    Threats to Validity and Limitations The main objective of this study is to analyze whether off-the-shelf LLMs are capable of replicating different facets of privacy policy analysis, both the end-to-end functional- ities as well as the intermediate tasks. Hence, our study design has a number of limitations that we address accord- ingly, as we describe belo...

  8. [8]

    For instance, Sunet al

    Related Work LLM and Privacy Policy Research: Privacy policy analysis research is increasingly leveraging LLMs to help users understand or analyze privacy policies, along various di- mensions [51], [31], [18], [39]. For instance, Sunet al. develop an LLM-based agent to act as a privacy policy expert to help users understand website privacy policies [47], ...

  9. [9]

    Conclusion In this paper, we conducted the first systematic evalua- tion of whether off-the-shelf LLMs can replace specialized privacy policy and compliance analysis tools, spanning three functionalities (contradiction detection, regulatory compli- ance analysis, and policy aggregation), two intermediate tasks (semantic role labeling and human manual anno...

  10. [10]

    https://gdpr-info.eu/art-13-gdpr/

    Gdpr article 13. https://gdpr-info.eu/art-13-gdpr/. Accessed June 2026

    2026

  11. [11]

    https://gdpr-info.eu/art-14-gdpr/

    Gdpr article 14. https://gdpr-info.eu/art-14-gdpr/. Accessed June 2026

    2026

  12. [12]

    https://texasattorneygeneral.gov/consumer-protection/ file-consumer-complaint/consumer-privacy-rights/ texas-data-privacy-and-security-act

    Texas data privacy and security act (tdpsa). https://texasattorneygeneral.gov/consumer-protection/ file-consumer-complaint/consumer-privacy-rights/ texas-data-privacy-and-security-act. Accessed June 2026

    2026

  13. [13]

    https://explore.usableprivacy.org/

    Usable privacy. https://explore.usableprivacy.org/. Accessed June 2026

    2026

  14. [14]

    Policypulse: Precision semantic role extraction for enhanced privacy policy com- prehension

    Andrick Adhikari, Sanchari Das, and Rinku Dewri. Policypulse: Precision semantic role extraction for enhanced privacy policy com- prehension. InNDSS, 2025

    2025

  15. [15]

    Machine learning algorithms for privacy policy classification: A comparative study

    Abdullah R Alshamsan and Shafique A Chaudhry. Machine learning algorithms for privacy policy classification: A comparative study. In 2022 IEEE 2nd International Conference on Software Engineering and Artificial Intelligence (SEAI), pages 214–219. IEEE, 2022

    2022

  16. [16]

    PolicyLint: Investigating Internal Privacy Policy Contradictions on Google Play

    Benjamin Andow, Samin Yaseer Mahmud, Wenyu Wang, Justin Whitaker, William Enck, Bradley Reaves, Kapil Singh, and Tao Xie. PolicyLint: Investigating Internal Privacy Policy Contradictions on Google Play. InProceedings of the USENIX Security Symposium, 2019

    2019

  17. [17]

    Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck

    Benjamin Andow, Samin Yaseer Mahmud, Justin Whitaker, William Enck, Bradley Reaves, Kapil Singh, and Serge Egelman. Actions Speak Louder than Words: Entity-Sensitive Privacy Policy and Data Flow Analysis with PoliCheck. InProceedings of the USENIX Security Symposium, 2020

    2020

  18. [18]

    Ac- tions speak louder than words: Entity-sensitive privacy policy and data flow analysis with policheck

    Benjamin Andow, Samin Yaseer Mahmud, Justin Whitaker, William Enck, Bradley Reaves, Kapil Singh, and Serge Egelman. Ac- tions speak louder than words: Entity-sensitive privacy policy and data flow analysis with policheck. In29th USENIX Security Symposium (USENIX Security 20), pages 985–1002. USENIX As- sociation, August 2020. URL: https://www.usenix.org/c...

    2020

  19. [19]

    Online artifacts for ”a systematic eval- uation of traditional privacy policy analysis tools against llms”

    Anonymous. Online artifacts for ”a systematic eval- uation of traditional privacy policy analysis tools against llms”. https://anonymous.4open.science/r/ privacy-and-compliance-analysis-tool-vs-llms-2B77/. Accessed June 2026

    2026

  20. [20]

    Breaux, Joel R

    Jaspreet Bhatia, Travis D. Breaux, Joel R. Reidenberg, and Thomas B. Norton. A Theory of Vagueness and Privacy Risk Perception. In Proceedings of the IEEE International Requirements Engineering Conference (RE), 2016

    2016

  21. [21]

    Sherman, Patrick Traynor, and Kevin Butler

    Jasmine Bowers, Bradley Reaves, Imani N. Sherman, Patrick Traynor, and Kevin Butler. Regulators, Mount Up! Analysis of Privacy Policies for Mobile Money Services. InProceedings of the Symposium on Usable Privacy and Security (SOUPS), 2017

    2017

  22. [22]

    Characterizing Security and Privacy Practices in Emerging Digital Credit Applications

    Jasmine Bowers, Imani N Sherman, Kevin Butler, and Patrick Traynor. Characterizing Security and Privacy Practices in Emerging Digital Credit Applications. InProceedings of the ACM Conference on Security and Privacy in Wireless and Mobile Networks (WiSec), 2019

    2019

  23. [23]

    Consistency analysis of data-usage purposes in mobile apps

    Duc Bui, Yuan Yao, Kang G Shin, Jong-Min Choi, and Junbum Shin. Consistency analysis of data-usage purposes in mobile apps. InProceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security, pages 2824–2843, 2021

    2021

  24. [24]

    Automat- ing governing knowledge commons and contextual integrity (gkc-ci) privacy policy annotations with large language models.arXiv preprint arXiv:2311.02192, 2023

    Jake Chanenson, Madison Pickering, and Noah Apthorpe. Automat- ing governing knowledge commons and contextual integrity (gkc-ci) privacy policy annotations with large language models.arXiv preprint arXiv:2311.02192, 2023

    Pith/arXiv arXiv 2023

  25. [25]

    A Large- Scale Evaluation of US Financial Institutions’ Standardized Privacy Notices.ACM Transactions on the Web, 2016

    Lorrie Faith Cranor, Pedro Giovanni Leon, and Blase Ur. A Large- Scale Evaluation of US Financial Institutions’ Standardized Privacy Notices.ACM Transactions on the Web, 2016

    2016

  26. [26]

    In32nd USENIX Security Symposium (USENIX Security 23), pages 1037–1054, 2023

    Hao Cui, Rahmadi Trimananda, Athina Markopoulou, and Scott Jordan.{PoliGraph}: Automated privacy policy analysis using knowl- edge graphs. In32nd USENIX Security Symposium (USENIX Security 23), pages 1037–1054, 2023

    2023

  27. [27]

    We Value Your Privacy

    Martin Degeling, Christine Utz, Christopher Lentzsch, Henry Hos- seini, Florian Schaub, and Thorsten Holz. We Value Your Privacy... Now Take Some Cookies: Measuring the GDPR’s Impact on Web Privacy. InProceedings of the ISOC Network and Distributed Systems Symposium (NDSS), 2018

    2018

  28. [28]

    ” you don’t need a university degree to comprehend data protection this way”: Llm-powered interactive privacy policy assessment

    Vincent Freiberger, Arthur Fleig, and Erik Buchmann. ” you don’t need a university degree to comprehend data protection this way”: Llm-powered interactive privacy policy assessment. InProceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–12, 2025

    2025

  29. [29]

    GDPR-info. Art. 9 gdpr - processing of special categories of personal data. https://gdpr-info.eu/art-9-gdpr/. Accessed May 2023

    2023

  30. [30]

    Creation and analysis of an international corpus of privacy laws

    Sonu Gupta, Geetika Gopi, Harish Balaji, Ellen Poplavska, Nora O’Toole, Siddhant Arora, Thomas Norton, Norman Sadeh, and Shomir Wilson. Creation and analysis of an international corpus of privacy laws. InProceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4092–4105, 2024

    2024

  31. [31]

    Bamberger, and Serge Egelman

    Catherine Han, Irwin Reyes, ´Alvaro Feal, Joel Reardon, Primal Wijesekera, Narseo Vallina-Rodriguez, Amit Elazari, Kenneth A. Bamberger, and Serge Egelman. The Price is (Not) Right: Comparing Privacy in Free and Paid Apps. InProceedings on Privacy Enhancing Technologies (PETS), 2020

    2020

  32. [32]

    Shin, and Karl Aberer

    Hamza Harkous, Kassem Fawaz, R ´emi Lebret, Florian Schaub, Kang G. Shin, and Karl Aberer. Polisis: Automated Analysis and Presentation of Privacy Policies Using Deep Learning. InProceedings of the USENIX Security Symposium, 2018

    2018

  33. [33]

    Polisis: Automated analysis and presentation of privacy policies using deep learning

    Hamza Harkous, Kassem Fawaz, R ´emi Lebret, Florian Schaub, Kang G Shin, and Karl Aberer. Polisis: Automated analysis and presentation of privacy policies using deep learning. In27th USENIX Security Symposium (USENIX Security 18), pages 531–548, 2018

    2018

  34. [34]

    Degpt: Optimizing decompiler output with llm

    Peiwei Hu, Ruigang Liang, and Kai Chen. Degpt: Optimizing decompiler output with llm. InNDSS, 2024

    2024

  35. [35]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, 2025

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.ACM Transactions on Information Systems, 43(2):1–55, 2025

    2025

  36. [36]

    Privacy Policies as Decision-Making Tools: An Evaluation of Online Privacy Notices

    Carlos Jensen and Colin Potts. Privacy Policies as Decision-Making Tools: An Evaluation of Online Privacy Notices. InProceedings of the ACM CHI Conference on Human Factors in Computing Systems (CHI), 2004

    2004

  37. [37]

    Understanding gdpr non- compliance in privacy policies of alexa skills in european market- places

    Song Liao, Mohammed Aldeen, Jingwen Yan, Long Cheng, Xiapu Luo, Haipeng Cai, and Hongxin Hu. Understanding gdpr non- compliance in privacy policies of alexa skills in european market- places. InProceedings of the ACM Web Conference 2024, pages 1081–1091, 2024

    2024

  38. [38]

    Towards automatic classification of privacy policy text.School of Computer Science Carnegie Mellon University, 2018

    Frederick Liu, Shomir Wilson, Peter Story, Sebastian Zimmeck, and Norman Sadeh. Towards automatic classification of privacy policy text.School of Computer Science Carnegie Mellon University, 2018

    2018

  39. [39]

    Have you been properly notified? automatic compliance analysis of privacy policy text with gdpr article 13

    Shuang Liu, Baiyang Zhao, Renjie Guo, Guozhu Meng, Fan Zhang, and Meishan Zhang. Have you been properly notified? automatic compliance analysis of privacy policy text with gdpr article 13. In Proceedings of the Web Conference 2021, pages 2154–2164, 2021

    2021

  40. [40]

    Smart Home Privacy Policies Demystified: A Study of Availability, Content, and Coverage

    Sunil Manandhar, Kaushal Kafle, Benjamin Andow, Kapil Singh, and Adwait Nadkarni. Smart Home Privacy Policies Demystified: A Study of Availability, Content, and Coverage. InProceedings of the 31st USENIX Security Symposium (USENIX’22), 2022

    2022

  41. [41]

    McDonald and Lorrie Faith Cranor

    Aleecia M. McDonald and Lorrie Faith Cranor. The Cost of Reading Privacy Policies.I/S Journal of Law and Policy for the Information Society (ISJLP), 4, 2008

    2008

  42. [42]

    On the ridiculousness of notice and consent: Con- tradictions in app privacy policies

    Ehimare Okoyomon, Nikita Samarin, Primal Wijesekera, Amit Elazari Bar On, Narseo Vallina-Rodriguez, Irwin Reyes, ´Alvaro Feal, Serge Egelman, et al. On the ridiculousness of notice and consent: Con- tradictions in app privacy policies. InWorkshop on Technology and Consumer Protection (ConPro 2019), in conjunction with the 39th IEEE Symposium on Security a...

    2019

  43. [43]

    On the Ridiculousness of Notice and Consent: Contradictions in App Privacy Policies

    Ehimare Okoyomon, Nikita Samarin, Primal Wijesekera, Amit Elazari Bar On, Narseo Vallina-Rodriguez, Irwin Reyes, ´Alvaro Feal, and Serge Egelman. On the Ridiculousness of Notice and Consent: Contradictions in App Privacy Policies. InWorkshop on Technology and Consumer Protection (ConPro), 2019

    2019

  44. [44]

    Morgan & Claypool San Rafael, 2011

    Martha Palmer, Daniel Gildea, and Nianwen Xue.Semantic role labeling. Morgan & Claypool San Rafael, 2011

    2011

  45. [45]

    A{NEW}{HOPE}: Contextual privacy policies for mobile applications and an approach toward automated generation

    Shidong Pan, Zhen Tao, Thong Hoang, Dawen Zhang, Tianshi Li, Zhenchang Xing, Xiwei Xu, Mark Staples, Thierry Rakotoarivelo, and David Lo. A{NEW}{HOPE}: Contextual privacy policies for mobile applications and an approach toward automated generation. In33rd USENIX Security Symposium (USENIX Security 24), pages 5699–5716, 2024

    2024

  46. [46]

    Is it a trap? a large-scale empirical study and comprehensive assessment of online automated privacy policy generators for mobile apps

    Shidong Pan, Dawen Zhang, Mark Staples, Zhenchang Xing, Jieshan Chen, Xiwei Xu, and Thong Hoang. Is it a trap? a large-scale empirical study and comprehensive assessment of online automated privacy policy generators for mobile apps. In33rd USENIX Security Symposium (USENIX Security 24), pages 5681–5698, 2024

    2024

  47. [47]

    Large language models meet nlp: A survey.Frontiers of Computer Science, 20(11):2011361, 2026

    Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. Large language models meet nlp: A survey.Frontiers of Computer Science, 20(11):2011361, 2026

    2026

  48. [48]

    Large language models: a new approach for privacy policy analysis at scale.arXiv preprint arXiv:2405.20900, 2024

    David Rodriguez, Ian Yang, Jose M Del Alamo, and Norman Sadeh. Large language models: a new approach for privacy policy analysis at scale.arXiv preprint arXiv:2405.20900, 2024

    arXiv 2024

  49. [49]

    Identifying the Provision of Choices in Privacy Policy Text

    Kanthashree Mysore Sathyendra, Shomir Wilson, Florian Schaub, Sebastian Zimmeck, and Norman Sadeh. Identifying the Provision of Choices in Privacy Policy Text. InProceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), 2017

    2017

  50. [50]

    Quan- tifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting

    Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quan- tifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting. In International Conference on Learning Representations, volume 2024, pages 25055–25083, 2024

    2024

  51. [51]

    A thorough examination of decoding methods in the era of llms

    Chufan Shi, Haoran Yang, Deng Cai, Zhisong Zhang, Yifan Wang, Yujiu Yang, and Wai Lam. A thorough examination of decoding methods in the era of llms. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024

    2024

  52. [52]

    The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism

    Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism. InProceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), 2025

    2025

  53. [53]

    Empowering users in digital privacy management through interactive llm-based agents

    Bolun Sun, Yifan Zhou, and Haiyun Jiang. Empowering users in digital privacy management through interactive llm-based agents. In International Conference on Learning Representations, volume 2025, pages 76740–76760, 2025

    2025

  54. [54]

    Large language models are in-context semantic reasoners rather than symbolic reasoners.arXiv preprint arXiv:2305.14825, 2023

    Xiaojuan Tang, Zilong Zheng, Jiaqi Li, Fanxu Meng, Song-Chun Zhu, Yitao Liang, and Muhan Zhang. Large language models are in-context semantic reasoners rather than symbolic reasoners.arXiv preprint arXiv:2305.14825, 2023

    Pith/arXiv arXiv 2023

  55. [55]

    The creation and analysis of a website privacy policy corpus

    Shomir Wilson, Florian Schaub, Aswarth Abhilash Dara, Frederick Liu, Sushain Cherivirala, Pedro Giovanni Leon, Mads Schaarup Andersen, Sebastian Zimmeck, Kanthashree Mysore Sathyendra, N Cameron Russell, et al. The creation and analysis of a website privacy policy corpus. InProceedings of the 54th Annual Meeting of the Association for Computational Lingui...

    2016

  56. [56]

    Policychecker: Ana- lyzing the gdpr completeness of mobile apps’ privacy policies

    Anhao Xiang, Weiping Pei, and Chuan Yue. Policychecker: Ana- lyzing the gdpr completeness of mobile apps’ privacy policies. In Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security, pages 3373–3387, 2023

    2023

  57. [57]

    Evaluating privacy policies under modern privacy laws at scale: An{LLM-Based} automated approach

    Qinge Xie, Karthik Ramakrishnan, and Frank Li. Evaluating privacy policies under modern privacy laws at scale: An{LLM-Based} automated approach. In34th USENIX Security Symposium (USENIX Security 25), pages 5797–5816, 2025

    2025

  58. [58]

    Can We Trust the Privacy Policies of Android Apps? InProceedings of the IEEE/I- FIP International Conference on Dependable Systems and Networks (DSN), 2016

    Le Yu, Xiapu Luo, Xule Liu, and Tao Zhang. Can We Trust the Privacy Policies of Android Apps? InProceedings of the IEEE/I- FIP International Conference on Dependable Systems and Networks (DSN), 2016

    2016

  59. [59]

    Privcaptcha: Interactive captcha to facilitate effective comprehension of app privacy policy

    Shuning Zhang, Xin Yi, Shixuan Li, Haobin Xing, and Hewu Li. Privcaptcha: Interactive captcha to facilitate effective comprehension of app privacy policy. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2025

    2025

  60. [60]

    A five-step workflow to manually annotate unstructured data into training dataset for natural language processing

    Yunshu Zhu, Ting Song, Zhenyu Zhang, Mengyang Yin, and Ping Yu. A five-step workflow to manually annotate unstructured data into training dataset for natural language processing. InMEDINFO 2023—The Future Is Accessible: Proceedings of the 19th World Congress on Medical and Health Informatics, pages 109–113. SAGE Publications 1 Oliver’s Yard, 55 City Road,...

    2023

  61. [61]

    Bellovin, and Joel Reidenberg

    Sebastian Zimmeck, Ziqi Wang, Lieyong Zou, Roger Iyengar, Bin Liu, Florian Schaub, Shomir Wilson, Norman Sadeh, Steven M. Bellovin, and Joel Reidenberg. Automated Analysis of Privacy Requirements for Mobile Apps. InProceedings of the ISOC Network and Distributed System Security Symposium (NDSS), 2017. Appendix

    2017

  62. [62]

    Figure 1 and 2 shows the data class level precision for Purpliance and PolicyPulse respectively

    Semantic Role Labeling Data Class Precision Data class level precision heatmaps for Purpliance and PolicyPulse across the four scored data classes (data, action,receiver, andpurpose), comparing each tool with LLM configurations under thinking mode. Figure 1 and 2 shows the data class level precision for Purpliance and PolicyPulse respectively. Figure 1. D...

  63. [63]

    Detailed(Prompt 2) LLM Example This appendix lists anexampleof the prompts used to instruct GPT-5.2 and Gemini-2.5-Pro in each functionality replication task (Sec. 4). In all prompts,<INPUT>is re- placed at runtime with the plain-text privacy policy under analysis. We providePrompt 2,i.e., detailed prompt, for the contradiction analysis task for PolicyLin...

  64. [64]

    9Statement B admits a practice (collect or share) for a SPECIFIC item that falls within that broad category (e.g ., Email Address)

    Logical Contradictions Identify pairs where: 7 8Statement A denies a practice (not_collect or not_share) for a BROAD category (e.g., Personal Information). 9Statement B admits a practice (collect or share) for a SPECIFIC item that falls within that broad category (e.g ., Email Address). 10 11

  65. [66]

    contradictions

    System and User Prompt Example (PolicyLint) The system-prompt experiment (see Section 7) splits the combinedPrompt 2 prompt into a fixedsystemmessage (role and JSON output schema) and a per-policyusermessage (contradiction criteria plus<INPUT>). System prompt TABLE 2. OPP-115CATEGORY-LEVEL METRICS VS.CONSOLIDATED (1.0)GROUND TRUTH. MICRO-AVERAGEDPRECISION...

  66. [67]

    6Statement B admits a practice (collect or share) for a SPECIFIC item that falls within that broad category (e.g ., Email Address)

    Logical Contradictions Identify pairs where: 4 5Statement A denies a practice (not_collect or not_share) for a BROAD category (e.g., Personal Information). 6Statement B admits a practice (collect or share) for a SPECIFIC item that falls within that broad category (e.g ., Email Address). 7 8

  67. [92]

    12Statement B denies a practice (not_collect or not_share) for a SPECIFIC item within that category

    Narrowing Definitions Identify pairs where: 10 11Statement A admits a practice (collect or share) for a BROAD category. 12Statement B denies a practice (not_collect or not_share) for a SPECIFIC item within that category. 13 14Input Privacy Policy: 15<INPUT> TABLE 3. OPP-115PER-HUMAN ANNOTATOR COMPARISON(THINKING MODE,CATEGORIES1AND3COMBINED). F1 (%)OF EAC...

  68. [122]

    contradictions

    Narrowing Definitions Identify pairs where: 13 14Statement A admits a practice (collect or share) for a BROAD category. 15Statement B denies a practice (not_collect or not_share) for a SPECIFIC item within that category. 16 17Input Privacy Policy: 18<INPUT> 19 20Output Format: JSON 21Provide the analysis in the following JSON structure: 22 23{ 24"contradi...