Listen to the Voices of Everyday Users: Democratizing Privacy Ratings for Sensitive Data Access in Mobile Apps
Pith reviewed 2026-05-08 03:18 UTC · model grok-4.3
The pith
Everyday users can rate mobile app data access to complement expert privacy audits.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that democratizing privacy assessment by letting everyday users evaluate the necessity of sensitive data access in apps is feasible. Using the DePRa prototype, which supplies contextual explanations, lets users select representative categories, rate accesses on an intuitive scale, and adjust ratings by risk preference, the approach captures user opinions effectively. Evaluations with 200 users demonstrate that these ratings differ from but can complement expert assessments while supporting scalable and inclusive privacy evaluation.
What carries the argument
DePRa, a participatory-design prototype that supplies contextual explanations of data uses, offers category-based selection, and includes preference-based rating adjustment to collect user privacy ratings.
Load-bearing premise
Everyday users' ratings accurately reflect the true appropriateness and necessity of data access without being significantly shaped by the explanations or selection tools shown to them.
What would settle it
A study that compares DePRa user ratings against actual privacy incidents or user complaints for the same apps over time, checking whether apps rated as having unnecessary access show measurably higher misuse rates.
Figures
read the original abstract
Mobile apps frequently request excessive data access, raising significant privacy concerns. While regulations like GDPR emphasize data minimization, they provide limited guidance on concretely defining and enforcing necessary data access. Existing regulatory mechanisms primarily rely on expert-driven audits that face challenges in scalability, neutrality, and alignment with user expectations. In this paper, we propose a novel paradigm--democratizing privacy assessment, inspired by prior work on user-centric privacy perceptions--which repositions users as active evaluators in the privacy auditing process, recognizing that user perceptions of data usage play a crucial role in assessing the appropriateness and necessity of data access. To operationalize this paradigm, we introduce DePRa, a prototype system developed through participatory design, featuring contextual explanation provision, category-based representative selection, an intuitive rating interface, and preference-based rating adjustment. We evaluated DePRa with 200 everyday mobile app users, analyzing how effectively it captures user opinions on sensitive data access, comparing their privacy ratings with expert assessments, and exploring risk preference-based score calibration. Our findings show the feasibility and promise of democratized privacy assessment, highlighting its potential to complement expert auditing and support inclusive privacy evaluation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a paradigm of 'democratizing privacy assessment' for mobile apps' sensitive data accesses. It introduces the DePRa prototype (developed via participatory design) with features including contextual explanations, category-based representative selection, an intuitive rating interface, and preference-based adjustment. A study with 200 everyday users is used to analyze how well DePRa captures user opinions, compare the resulting privacy ratings against expert assessments, and explore risk-preference calibration; the authors conclude that the approach demonstrates feasibility and promise as a complement to expert auditing.
Significance. If the empirical results hold after addressing interface-bias concerns, the work could meaningfully scale privacy evaluation beyond expert-only audits, improve alignment with user expectations under data-minimization regulations such as GDPR, and support more inclusive auditing practices.
major comments (1)
- [Evaluation] Evaluation section (user-study description): the manuscript reports no ablation study or control condition that removes or varies DePRa's explanatory text, representative-selection mechanism, or preference-adjustment feature. Without such controls, the observed alignment between user ratings and expert assessments cannot be attributed to independent user judgment rather than interface steering, directly undermining the central feasibility claim that DePRa-collected ratings reflect authentic everyday-user perceptions suitable for complementing expert audits.
minor comments (1)
- [Abstract] Abstract: the summary omits any mention of study design details, statistical methods, or quantitative comparison metrics between user and expert ratings, making it difficult for readers to gauge the strength of the empirical support at first reading.
Simulated Author's Rebuttal
We thank the referee for the constructive feedback and recommendation for major revision. We address the evaluation concern below by explaining the rationale for our integrated study design and outlining targeted revisions to improve transparency.
read point-by-point responses
-
Referee: [Evaluation] Evaluation section (user-study description): the manuscript reports no ablation study or control condition that removes or varies DePRa's explanatory text, representative-selection mechanism, or preference-adjustment feature. Without such controls, the observed alignment between user ratings and expert assessments cannot be attributed to independent user judgment rather than interface steering, directly undermining the central feasibility claim that DePRa-collected ratings reflect authentic everyday-user perceptions suitable for complementing expert audits.
Authors: We acknowledge that the study evaluates the complete DePRa system without ablating features such as contextual explanations, category-based selection, or preference adjustment. This design choice stems from the participatory design process, in which everyday users identified these elements as essential for enabling non-experts to understand data access implications and provide informed ratings. Removing them (e.g., via a no-explanation control) would likely yield uninformed or random responses rather than authentic perceptions, undermining the goal of democratizing assessment. The observed alignment with expert ratings therefore reflects the practical utility of the full user-centered prototype. Nevertheless, we agree that this limits isolation of individual feature effects and potential interface steering. We will add a dedicated 'Limitations' subsection to the Evaluation section (Section 5) that explicitly discusses the absence of control conditions, notes the possibility of steering, and outlines planned future controlled experiments to vary features independently. This partial revision will qualify our feasibility claims while preserving the contribution of demonstrating an integrated, accessible tool. revision: partial
Circularity Check
No circularity: empirical user study with no derivations or self-referential reductions
full rationale
The paper introduces DePRa via participatory design and evaluates it through a 200-user study that collects ratings, compares them to expert assessments, and explores calibration. All central claims (feasibility, promise for complementing audits) rest on these empirical observations rather than any equations, first-principles derivations, fitted parameters renamed as predictions, or load-bearing self-citations. No step reduces by construction to its own inputs; the design features (explanations, selection) are explicitly part of the interface being tested, not hidden assumptions that force the outcome. This is a standard non-circular empirical contribution.
Axiom & Free-Parameter Ledger
axioms (1)
- domain assumption User perceptions of data usage play a crucial role in assessing the appropriateness and necessity of data access.
Reference graph
Works this paper leans on
-
[1]
Permission-greedy apps delayed Android 6 upgrade so they could har- vest more user data
2019. Permission-greedy apps delayed Android 6 upgrade so they could har- vest more user data. https://www.zdnet.com/article/permission-greedy-apps- delayed-android-6-upgrade-so-they-could-harvest-more-user-data/
2019
-
[2]
Protect Your Privacy From the Apps on Your Phone
2019. Protect Your Privacy From the Apps on Your Phone. https: //www.consumerreports.org/electronics-computers/privacy/protect-your- privacy-from-the-apps-on-your-phone-a1049648633/
2019
-
[3]
Android financial apps too greedy for permissions
2023. Android financial apps too greedy for permissions. https://cybernews.com/ security/android-financial-apps-greedy-for-permissions/
2023
-
[4]
Greedy apps collect more information than they should
2023. Greedy apps collect more information than they should. https://betanews. com/2023/10/27/greedy-apps-collect-more-information-than-they-should/
2023
-
[5]
BERTopic
2024. BERTopic. https://maartengr.github.io/BERTopic/index.html
2024
-
[6]
Google-Play-Scraper
2024. Google-Play-Scraper. https://github.com/JoMingyu/google-play-scraper
2024
-
[7]
Artifact Availability
2025. Artifact Availability. https://anonymous.4open.science/r/ DemocratizePrivacyRating-17E2
2025
-
[8]
Prolific: Easily collect high-quality data from real people
2025. Prolific: Easily collect high-quality data from real people. https://www. prolific.com/
2025
-
[9]
Third-Party Library Permission Piggybacking in Android Apps
2025. Third-Party Library Permission Piggybacking in Android Apps. https://blog.appicaptor.com/2025/02/27/third-party-library-permission- piggybacking-in-android-apps/
2025
-
[10]
Sadiq Aliyu, Sushmita Khan, Aminata N Mbodj, Oluwafemi Osho, Lingyuan Li, Bart Knijnenburg, and Mauro Cherubini. 2024. Participatory Design to Ad- dress Disclosure-Based Cyberbullying. InProceedings of the 2024 ACM Designing Interactive Systems Conference. 1547–1565
2024
-
[11]
Kevin Allix, Tegawendé F Bissyandé, Jacques Klein, and Yves Le Traon. 2016. Androzoo: Collecting millions of android apps for the research community. In 2016 IEEE/ACM 13th Working Conference on Mining Software Repositories (MSR). IEEE, 468–471
2016
-
[12]
Hazim Almuhimedi, Florian Schaub, Norman Sadeh, Idris Adjerid, Alessandro Acquisti, Joshua Gluck, Lorrie Faith Cranor, and Yuvraj Agarwal. 2015. Your Loca- tion has been Shared 5,398 Times! A Field Study on Mobile App Privacy Nudging. InProceedings of the 33rd Annual ACM Conference on Human Factors in Com- puting Systems(Seoul, Republic of Korea)(CHI ’15)...
-
[13]
Apple. 2025. Privacy on iOS. https://developer.apple.com/design/human- interface-guidelines/privacy
2025
-
[14]
Henrik Axelborn and John Berggren. 2023. Topic Modeling for Customer Insights: A Comparative Analysis of LDA and BERTopic in Categorizing Customer Calls
2023
-
[15]
Michael Backes, Sven Bugiel, and Erik Derr. 2016. Reliable Third-Party Library Detection in Android and its Security Applications. InProceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security(Vienna, Austria)(CCS ’16). Association for Computing Machinery, New York, NY, USA, 356–367. doi:10.1145/2976749.2978333
-
[16]
Susanne Barth, Dan Ionita, and Pieter Hartel. 2022. Understanding online pri- vacy—a systematic review of privacy visualizations and privacy by design guide- lines.ACM Computing Surveys (CSUR)55, 3 (2022), 1–37
2022
-
[17]
Kerstin Bongard-Blanchy, Jean-Louis Sterckx, Arianna Rossi, Verena Distler, Salvador Rivas, and Vincent Koenig. 2022. An (Un) Necessary Evil-Users’(Un) Certainty about Smartphone App Permissions and Implications for Privacy Engi- neering. In2022 IEEE European Symposium on Security and Privacy Workshops (EuroS&PW). IEEE, 01–08
2022
-
[18]
Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative Research in Psychology3, 2 (2006), 77–101
2006
-
[19]
Wenjuan Bu, Hui Shu, Fei Kang, Qian Hu, and Yuntian Zhao. 2023. Software Subclassification Based on BERTopic-BERT-BiLSTM Model.Electronics12, 18 (2023), 3798
2023
-
[20]
Kai Chih Chang, Razieh Nokhbeh Zaeem, and K Suzanne Barber. 2020. A frame- work for estimating privacy risk scores of mobile apps. InInternational Conference on Information Security. Springer, 217–233
2020
-
[21]
European Commission. 2022. General Data Protection Regulation. https:// commission.europa.eu/law/law-topic/data-protection_en
2022
-
[22]
European Commission. 2025. Art. 5 GDPR - Principles relating to processing of personal data. https://gdpr.eu/article-5-how-to-process-personal-data/
2025
-
[23]
DigiChina. 2021. Personal Information Protection Law of the People’s Republic of China. https://digichina.stanford.edu/work/translation-personal-information- protection-law-of-the-peoples-republic-of-china-effective-nov-1-2021/
2021
-
[24]
Roman Egger and Joanne Yu. 2022. A topic modeling comparison between lda, nmf, top2vec, and bertopic to demystify twitter posts.Frontiers in sociology7 (2022), 886498
2022
-
[25]
Houda Elmimouni, Eric P. S. Baumer, and Andrea Forte. 2024. On Being an Expert: Habitus as a Lens for Understanding Privacy Expertise.Proc. ACM Hum.-Comput. Interact.8, CSCW1, Article 102 (April 2024), 25 pages. doi:10.1145/3637379
-
[26]
Adrienne Porter Felt, Elizabeth Ha, Serge Egelman, Ariel Haney, Erika Chin, and David Wagner. 2012. Android permissions: user attention, comprehension, and behavior. InProceedings of the Eighth Symposium on Usable Privacy and Security (Washington, D.C.)(SOUPS ’12). Association for Computing Machinery, New York, NY, USA, Article 3, 14 pages. doi:10.1145/23...
-
[27]
Alisa Frik and Alexia Gaudeul. 2020. A measure of the implicit value of privacy under risk.Journal of Consumer Marketing37, 4 (2020), 457–472
2020
-
[28]
Google. 2025. Permissions on Android. https://developer.android.com/guide/ topics/permissions/overview
2025
-
[29]
Alessandra Gorla, Ilaria Tavecchia, Florian Gross, and Andreas Zeller. 2014. Checking app behavior against app descriptions. InProceedings of the 36th inter- national conference on software engineering. 1025–1035. Listen to the Voices of Everyday Users: Democratizing Privacy Ratings for Sensitive Data Access in Mobile Apps Conference’17, July 2017, Washin...
2014
-
[30]
Judith Gregory. 2003. Scandinavian Approaches to Participatory Design. https: //api.semanticscholar.org/CorpusID:14198877
2003
-
[31]
Asma Hamed and Hella Kaffel Ben Ayed. 2016. Privacy risk assessment and users’ awareness for mobile apps permissions. In2016 IEEE/ACS 13th International Conference of Computer Systems and Applications (AICCSA). IEEE, 1–8
2016
-
[32]
Kotaro Hara, Christine Chan, and Jon E Froehlich. 2016. The design of assistive location-based technologies for people with ambulatory disabilities: A formative study. InProceedings of the 2016 CHI conference on human factors in computing systems. 1757–1768
2016
-
[33]
ICCL. 2021. Europe’s Enforcement Paralysis: 2021 GDPR Report. https://www.iccl.ie/wp-content/uploads/2021/09/Europes-enforcement- paralysis-2021-ICCL-report-on-GDPR-enforcement.pdf
2021
-
[34]
Qatrunnada Ismail, Tousif Ahmed, Apu Kapadia, and Michael K Reiter. 2015. Crowdsourced exploration of security configurations. InProceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. 467–476
2015
-
[35]
Daniel Kahneman and Amos Tversky. 2013. Prospect theory: An analysis of decision under risk. InHandbook of the fundamentals of financial decision making: Part I. World Scientific, 99–127
2013
-
[36]
Jina Kang, Hyoungshick Kim, Yun Gyung Cheong, and Jun Ho Huh. 2015. Visual- izing privacy risks of mobile applications through a privacy meter. InInformation Security Practice and Experience: 11th International Conference, ISPEC 2015, Beijing, China, May 5-8, 2015, Proceedings. Springer, 548–558
2015
-
[37]
Mark J Keith, Jeffry S Babb, and Paul Benjamin Lowry. 2014. A longitudinal study of information privacy on mobile devices. In2014 47th Hawaii International Conference on System Sciences. IEEE, 3149–3158
2014
-
[38]
Patrick Gage Kelley, Lorrie Faith Cranor, and Norman Sadeh. 2013. Privacy as part of the app decision-making process. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems(Paris, France)(CHI ’13). Association for Computing Machinery, New York, NY, USA, 3393–3402. doi:10.1145/2470654. 2466466
-
[39]
Nabila Khodeir and Fatma Elghannam. 2024. Efficient topic identification for urgent MOOC Forum posts using BERTopic and traditional topic modeling tech- niques.Education and Information Technologies(2024), 1–27
2024
-
[40]
Nakyoung Kim, Hyeontaek Oh, and Jun Kyun Choi. 2023. A privacy scoring framework: Automation of privacy compliance and risk evaluation with standard indicators.Journal of King Saud University-Computer and Information Sciences 35, 1 (2023), 514–525
2023
-
[41]
Konrad Kollnig, Reuben Binns, Pierre Dewitte, Max Van Kleek, Ge Wang, Daniel Omeiza, Helena Webb, and Nigel Shadbolt. 2021. A fait accompli? an empirical study into the absence of consent to third-party tracking in android apps. In Proceedings of the Seventeenth USENIX Conference on Usable Privacy and Security (SOUPS’21). USENIX Association, USA, Article ...
2021
-
[42]
Klaus Krippendorff. 2011. Computing Krippendorff’s alpha-reliability. (2011)
2011
-
[43]
Jabari Kwesi, Jiaxun Cao, Riya Manchanda, and Pardis Emami-Naeini. 2025. Exploring User Security and Privacy Attitudes and Concerns Toward the Use of {General-Purpose} {LLM} Chatbots for Mental Health. In34th USENIX Security Symposium (USENIX Security 25). 6007–6024
2025
-
[44]
CMS Law. 2023. GDPR Enforcement Tracker Report. https://cms.law/en/media/ international/files/publications/publications/gdpr-enforcement-tracker-report- may-2023
2023
-
[45]
Guocheng Liao, Xu Chen, and Jianwei Huang. 2019. Prospect theoretic analysis of privacy-preserving mechanism.IEEE/ACM Transactions on Networking28, 1 (2019), 71–83
2019
-
[46]
Jialiu Lin, Shahriyar Amini, Jason I Hong, Norman Sadeh, Janne Lindqvist, and Joy Zhang. 2012. Expectation and purpose: understanding users’ mental models of mobile app privacy through crowdsourcing. InProceedings of the 2012 ACM conference on ubiquitous computing. 501–510
2012
-
[47]
Jialiu Lin, Bin Liu, Norman Sadeh, and Jason I Hong. 2014. Modeling {Users’} mobile app privacy preferences: restoring usability in a sea of permission settings. In10th Symposium On Usable Privacy and Security (SOUPS 2014). 199–212
2014
- [48]
-
[49]
Trung Tin Nguyen, Michael Backes, Ninja Marnau, and Ben Stock. 2021. Share first, ask later (or never?) studying violations of {GDPR’s} explicit consent in android apps. In30th USENIX Security Symposium (USENIX Security 21). 3667– 3684
2021
-
[50]
Helen Nissenbaum. 2004. Privacy as contextual integrity.Wash. L. Rev.79 (2004), 119
2004
-
[51]
OAG. 2022. California Consumer Privacy Act. https://oag.ca.gov/privacy/ccpa
2022
-
[52]
Leilei Qu, Cheng Wang, Ruojin Xiao, Jianwei Hou, Wenchang Shi, and Bin Liang
-
[53]
InExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems
Towards better security decisions: applying prospect theory to cyberse- curity. InExtended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems. 1–6
2019
-
[54]
Durity, and Lorrie Faith Cranor
Florian Schaub, Rebecca Balebako, Adam L. Durity, and Lorrie Faith Cranor. 2015. A design space for effective privacy notices. InProceedings of the Eleventh USENIX Conference on Usable Privacy and Security(Ottawa, Canada)(SOUPS ’15). USENIX Association, USA, 1–17
2015
-
[55]
Simmons. 2021. Ireland’s balance between Big Tech and data privacy. https://www.simmons-simmons.com/en/publications/ ckucpnrme21dy0a42mwuhhhae/ireland-s-balance-between-big-tech-and- data-privacy
2021
-
[56]
Qiurong Song, Yanlai Wu, Rie Helene (Lindy) Hernandez, Yao Li, Yubo Kou, and Xinning Gui. 2025. Understanding Users’ Perception of Personally Identifiable Information. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Association for Computing Machinery, New York, NY, USA, Article 240, 24 pages. doi:10.1145/3706598.3713783
-
[57]
syrenis. 2024. Privacy policies: Is anyone reading them? https://syrenis.com/ resources/blog/privacy-policies-is-anyone-reading-them/
2024
-
[58]
Mohammad Tahaei, Ruba Abu-Salma, and Awais Rashid. 2023. Stuck in the Permissions With You: Developer & End-User Perspectives on App Permissions & Their Privacy Ramifications. InProceedings of the 2023 CHI Conference on Human Factors in Computing Systems(Hamburg, Germany)(CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 168, 24 pa...
-
[59]
Junwei Tang, Ruixuan Li, Hongmu Han, Heng Zhang, and Xiwu Gu. 2017. De- tecting permission over-claim of android applications with static and semantic analysis approach. In2017 IEEE Trustcom/BigDataSE/ICESS. IEEE, 706–713
2017
-
[60]
Ehsan Ul Haque and Mohammad Maifi Hasan Khan. 2025. Investigating Users’ Decision-making for Data Privacy Controls in the Context of Internet of Things (IoT) Devices Using an Incentive-compatible Lottery Study. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI ’25). Asso- ciation for Computing Machinery, New York, NY, USA...
-
[61]
Haoyu Wang, Yuanchun Li, Yao Guo, Yuvraj Agarwal, and Jason I Hong. 2017. Understanding the purpose of permission use in mobile apps.ACM Transactions on Information Systems (TOIS)35, 4 (2017), 1–40
2017
-
[62]
Liu Wang, Dong Wang, Shidong Pan, Zheng Jiang, Haoyu Wang, and Yi Wang
-
[63]
In2025 IEEE Symposium on Security and Privacy (SP)
A big step forward? a user-centric examination of ios app privacy report and enhancements. In2025 IEEE Symposium on Security and Privacy (SP). IEEE, 4210–4228
-
[64]
WIRED. 2022. How GDPR Is Failing. https://www.wired.com/story/gdpr-2022/
2022
-
[65]
Shao Yang, Yuehan Wang, Yuan Yao, Haoyu Wang, Yanfang Ye, and Xusheng Xiao
-
[66]
InProceedings of the 44th International Conference on Software Engineering
Describectx: context-aware description synthesis for sensitive behaviors in mobile apps. InProceedings of the 44th International Conference on Software Engineering. 685–697
-
[67]
Yang Yang, Xuehui Du, and Zhi Yang. 2021. Pradroid: Privacy risk assessment for android applications. In2021 IEEE 5th International Conference on Cryptography, Security and Privacy (CSP). IEEE, 90–95
2021
-
[68]
Kutsal Yesilkagit. 2011. Institutional compliance, European networks of regulation and the bureaucratic autonomy of national regulatory authorities.Journal of European Public Policy18, 7 (2011), 962–979
2011
-
[69]
Lu Zhou, Chengyongxiao Wei, Tong Zhu, Guoxing Chen, Xiaokuan Zhang, Suguo Du, Hui Cao, and Haojin Zhu. 2023. {POLICYCOMP}: Counterpart Comparison of Privacy Policies Uncovers Overbroad Personal Data Collection Practices. In 32nd USENIX Security Symposium (USENIX Security 23). 1073–1090. Conference’17, July 2017, Washington, DC, USA Liu Wang, Tianshu Zhou,...
2023
-
[70]
Background Information. •Prolific ID: (Required) •Daily time spent using mobile apps: (Required) Types of apps you frequently use: (Select at least one) □Social□Shopping □Weather□Business □Education□Maps and Navigation □Music and Audio□Health and Fitness □Others
-
[71]
Mobile Privacy Preferences. A. Sensitivity of Data Types Which of the following user data types do you consider sensitive? (Select at least one) □Name and Contact Information□Geographic Location □Call Records□App Usage Records □Biometric Data□Health Data □Bank Transaction Data□Photos Other sensitive data types (if any): B. Awareness of Privacy Rights Do y...
-
[72]
Which option would you choose?(Required) ⃝A
You participated in a lottery. Which option would you choose?(Required) ⃝A. Guaranteed $90 reward ⃝B. 95% chance to win $100, 5% chance to win nothing
-
[73]
Which option would you choose?(Required) ⃝A
You participated in a lucky draw. Which option would you choose?(Required) ⃝A. Guaranteed $5 reward ⃝B. 5% chance to win $100, 95% chance to win nothing
-
[74]
Which option would you choose?(Required) ⃝A
You need to pay a fee to participate in a game. Which option would you choose?(Required) ⃝A. Pay $90 for sure ⃝B. 95% chance to pay $100, 5% chance to pay nothing
-
[75]
Which option would you choose?(Required) ⃝A
You were told you might need to pay a fine. Which option would you choose?(Required) ⃝A. Pay $5 for sure ⃝B. 5% chance to pay $100, 95% chance to pay nothing
-
[76]
User Preferences and Attitudes.Which type of app do you prefer?(Required) ⃝Free apps with extensive permissions ⃝Paid apps without personal data collection What are your thoughts on mobile privacy protection?
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.