{"id":"bc1efd04-fddc-43c9-8cb6-662d4368d1bf","arxiv_id":"2606.02960","paper_version":1,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Empirical study of Android apps yields 31 coding guidelines that reduce data minimization violations in both human-written and LLM-generated code.","lead":"This paper analyzed 1,114 open-source Android apps to identify ten common data minimization scenarios, then examined 9,875 real apps to create 31 coding guidelines. A smart generalist might read it to see how code-level rules can improve app privacy and fix problems in AI coding tools.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Representativeness of the 10 scenarios from 1,114 open-source apps for distilling generalizable guidelines","rationale":"The reader's weakest assumption directly identifies the load-bearing step: the leap from the formative study's 10 scenarios to generalizable guidelines. No other internal inconsistency (e.g., in the LLM prompting protocol) appears more central given the abstract and reader's summary; the proposed test would falsify or confirm that leap without requiring full re-analysis.","tokens_in":1710,"tokens_out":308,"duration_ms":10629,"concrete_test":"Randomly sample 200 apps from the 9,875-APK corpus, manually classify their data-handling code against the 10 scenarios, and compute the fraction of apps whose risky patterns fall outside the 10 scenarios; if >15% of apps exhibit uncovered patterns, rerun the LLM evaluation with additional scenarios and measure whether guideline effectiveness drops.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the 31 guidelines eliminate risky practices in LLMs rests on the 10 scenarios being sufficiently representative. These scenarios come from a formative study of open-source apps; the subsequent 9,875-APK analysis is used only to distill guidelines, not to validate coverage. If the open-source sample under-represents closed-source or commercial patterns (e.g., in permission handling or third-party SDK usage), the guidelines may miss prevalent risky practices, weakening the claim that they fix LLM outputs across real-world Android development.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper conducts a formative study on 1,114 open-source Android apps to identify ten recurring data minimization scenarios across five data-handling stages, performs a large-scale analysis of 9,875 real-world APKs to distill 31 actionable coding guidelines, and evaluates state-of-the-art LLMs, finding that they reproduce data minimization-risky practices which are eliminated when the guidelines are incorporated.","tokens_in":1819,"tokens_out":498,"duration_ms":24086,"significance":"If the methodological gaps are addressed, the work would provide concrete code-level guidance for data minimization in Android development and demonstrate a practical way to improve LLM-assisted code generation for privacy compliance. The scale of the APK analysis combined with the LLM evaluation represents a useful empirical contribution to software engineering and privacy research.","major_comments":[{"comment":"Formative study section: no details are provided on inter-rater reliability, validation of scenario detection, or agreement metrics for deriving the ten scenarios from the 1,114 open-source apps; this directly affects the reliability of the scenarios used to frame the subsequent guideline distillation.","section":"Formative study methodology"},{"comment":"Large-scale APK analysis section: the description of the 9,875-APK inspection provides no information on how false positives are avoided or mitigated when detecting data minimization scenarios, which is load-bearing for the accuracy and soundness of the 31 distilled guidelines.","section":"Large-scale analysis"},{"comment":"Guideline derivation and evaluation: the ten scenarios originate from an open-source sample, yet the 9,875-APK analysis is used only to distill guidelines rather than to validate coverage or representativeness for closed-source/commercial apps (e.g., permission handling or third-party SDK patterns); this leaves the central claim that the guidelines eliminate risky practices across real-world Android development on an untested assumption.","section":"Guideline applicability and LLM evaluation"}],"minor_comments":[{"comment":"The abstract could more explicitly state the number and identities of the LLMs evaluated and the precise risky practices observed before claiming elimination across all models.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The low soundness score and representativeness concern in the reader notes align with the major comments; revisions addressing validation would strengthen the manuscript for a cs.SE venue."},"author_rebuttal":{"model":"grok-4.3","summary":"We are grateful for the referee's constructive feedback, which identifies key areas where additional methodological transparency would strengthen the paper. We address each major comment below and indicate the corresponding revisions.","responses":[{"response":"We thank the referee for this observation. The formative study was conducted by two authors who independently coded a random subset of 200 apps from the 1,114-app corpus, iteratively refined the scenario definitions through discussion, and validated the final ten scenarios against the full set. Although formal agreement metrics were not reported in the original manuscript, we agree that documenting the process is essential. We will add a dedicated paragraph in the formative study section describing the coding procedure, the pre-discussion agreement rate, and how discrepancies were resolved to derive the scenarios.","revision_made":"yes","referee_comment":"[Formative study methodology] Formative study section: no details are provided on inter-rater reliability, validation of scenario detection, or agreement metrics for deriving the ten scenarios from the 1,114 open-source apps; this directly affects the reliability of the scenarios used to frame the subsequent guideline distillation."},{"response":"We agree that explicit details on false-positive mitigation are required. Our analysis combined automated detection via static analysis scripts with manual verification on a stratified random sample of 500 APKs to compute an empirical false-positive rate and refine the detection rules before guideline distillation. We will expand the large-scale analysis section with a new subsection that reports the sampling strategy, the observed false-positive rate from manual review, and how these checks informed the final set of 31 guidelines.","revision_made":"yes","referee_comment":"[Large-scale analysis] Large-scale APK analysis section: the description of the 9,875-APK inspection provides no information on how false positives are avoided or mitigated when detecting data minimization scenarios, which is load-bearing for the accuracy and soundness of the 31 distilled guidelines."},{"response":"The ten scenarios were derived from open-source apps to enable source-level inspection, but the 9,875-APK corpus consists primarily of closed-source commercial applications and was used both to quantify scenario prevalence and to surface the concrete code patterns that became the 31 guidelines. Patterns involving permission handling and third-party SDKs were directly observed in the APK dataset. We will revise the discussion and threats-to-validity sections to clarify this distinction, explicitly state the generalizability claim, and acknowledge that a dedicated closed-source validation corpus was not employed. This addresses the concern without altering the empirical grounding of the guidelines.","revision_made":"partial","referee_comment":"[Guideline applicability and LLM evaluation] Guideline derivation and evaluation: the ten scenarios originate from an open-source sample, yet the 9,875-APK analysis is used only to distill guidelines rather than to validate coverage or representativeness for closed-source/commercial apps (e.g., permission handling or third-party SDK patterns); this leaves the central claim that the guidelines eliminate risky practices across real-world Android development on an untested assumption."}],"tokens_in":1372,"tokens_out":599,"duration_ms":23844,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The paper's main move is turning a formative study of 1,114 open-source Android apps into ten data-minimization scenarios, then using those to guide a large-scale scan of 9,875 APKs and pull out 31 concrete coding guidelines. It also tests current LLMs on code generation and reports that the models repeat the risky patterns seen in the wild, but adding the guidelines removes the problems across the models they tried.\n\nWhat stands out is the shift from policy-level checks to code-level advice that developers or LLM prompts could actually use. The scale of the APK analysis and the before-after LLM results give the work a practical edge that most privacy papers lack.\n\nThe soft spot is exactly the one the stress-test flags. The scenarios come only from open-source apps; the APK analysis then applies them but does not test whether the ten scenarios miss common closed-source or SDK-heavy patterns. If commercial apps handle permissions or third-party data differently, the guidelines could leave gaps. The abstract gives no numbers on inter-rater checks or false-positive rates in the scenario detection, so the central claim that the guidelines are broadly effective is still provisional.\n\nThis is the kind of paper that belongs in a reading group for people working on mobile privacy or LLM-assisted development. It is not yet a finished result, but the empirical pipeline and the LLM experiment are worth referee time. I would send it out for review rather than desk-reject; the authors need to show that the open-source sample is representative enough for the guidelines to travel.","headline":"The 31 guidelines and LLM fix are the real output here, but they rest on whether 10 scenarios from 1,114 open-source apps cover real-world Android patterns.","tokens_in":2283,"tokens_out":392,"would_cite":false,"duration_ms":13027,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Thirty-one code guidelines from 11,000 Android apps steer both developers and LLMs away from data minimization violations.","keywords":["data minimization","Android apps","privacy compliance","coding guidelines","LLM code generation","empirical study","APK analysis","developer practices"],"falsifier":"Discovery of many real-world apps that systematically violate data minimization in ways outside the ten identified scenarios would show the guidelines do not cover the space.","tokens_in":2612,"feed_emoji":"📱","tokens_out":587,"duration_ms":20440,"temperature":0.7,"pith_summary":"The authors first study 1,114 open-source Android apps to isolate ten recurring data minimization scenarios that occur across five stages of data handling. They then extract 31 concrete coding guidelines and confirm their relevance by inspecting 9,875 real-world APKs. When these guidelines are supplied to state-of-the-art language models, the models stop reproducing the risky patterns they otherwise copy from existing code. A sympathetic reader cares because the work supplies ready-to-use, code-level steps that address privacy rules at the point where apps are actually written or generated.","feed_headline":"31 guidelines fix data minimization risks copied by LLMs","feed_subtitle":"Analysis of 11k Android apps yields rules that stop models from reproducing real-world privacy violations.","key_machinery":"The ten recurring data minimization scenarios across five data-handling stages, used to derive the 31 actionable coding guidelines.","core_discovery":"Empirical examination of code in more than eleven thousand Android applications reveals ten common data minimization scenarios; from these the authors derive thirty-one actionable coding guidelines. State-of-the-art LLMs trained on real-world code reproduce the same risky patterns, yet the guidelines eliminate the issues in every model tested.","pith_inferences":["The guidelines might transfer to other mobile platforms that use similar data-handling stages.","Embedding the rules in static analysis tools could catch violations during routine code review.","Widespread use could lower the aggregate amount of personal data collected by mobile apps over time.","New data categories such as health or location records could be checked against the same five-stage framework."],"forward_implications":["Apps that follow the 31 guidelines satisfy data minimization at the implementation level.","LLM-generated Android code becomes free of the identified risky practices once the guidelines are incorporated.","Privacy regulation can be met by changing code practices rather than only updating privacy policies.","The same guidelines serve both human developers and AI-assisted programming workflows."],"fun_headline_variants":["31 guidelines stop LLMs copying Android data minimization risks","11k app analysis yields rules fixing LLM privacy violations","Ten scenarios from 11k apps produce 31 LLM risk guidelines","Study shows LLMs repeat Android risks but guidelines fix them all"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The ten scenarios found in the open-source sample capture the main patterns present in the larger population of real-world Android apps.","fun_headline_variants_meta":{"raw":{"variants":["31 guidelines stop LLMs copying Android data minimization risks","11k app analysis yields rules fixing LLM privacy violations","Ten scenarios from 11k apps produce 31 LLM risk guidelines","Study shows LLMs repeat Android risks but guidelines fix them all"]},"model":"grok-4.3","cost_usd":0.005001,"raw_usage":{"total_tokens":2422,"prompt_tokens":628,"num_sources_used":0,"completion_tokens":66,"cost_in_usd_ticks":50012000,"prompt_tokens_details":{"text_tokens":628,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1728,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":628,"tokens_out":66,"duration_ms":13410,"temperature":1.0,"reasoning_tokens":1728,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T13:13:49.708274+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Discovery of many real-world apps that systematically violate data minimization in ways outside the ten identified scenarios would show the guidelines do not cover the space.","supporting_citations":[],"review_version":1}