{"id":"432947b1-ed67-4b07-8b7e-619e278c4f7b","arxiv_id":"2507.07927","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":8.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Most Android apps that collect sensitive data still rely on software key storage, and the first large-scale benchmarks show secure-element key operations are far too slow for anything but small payloads.","lead":"KeyDroid analyzed roughly 490,000 Android apps and found that most apps collecting sensitive data do not use the phone's hardware-backed key storage, while only about 5% use the strongest secure-element option. It also publishes the first large set of benchmark timings showing that trusted-execution-environment key storage is fast enough for most apps, while secure-element storage is dramatically slower.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Play Store data safety labels form a fragile denominator for the headline adoption rates; a sensitivity check against an independent sensitive-data proxy is needed.","rationale":"The reader's weakest assumption correctly identifies the data safety labels as the most fragile link in the chain from measurements to headline claims. I agree with that identification after reading the paper: the 56.3% and 5.03% figures are explicitly computed over the 159,241 apps that self-report sensitive-data collection via Play Store labels, and §A.2 documents both the off-device-transmission definition and the 74.47% submission rate. The paper is careful to say 'self-reporting' in the abstract, which limits the literal claim, but the Discussion and Conclusion generalize to apps that 'collect potentially sensitive user data', and the policy narrative ('despite industry-wide initiatives, adoption remains low') requires the denominator to be a meaningful proxy for the population of interest. The paper's counter-argument — that apps hiding local sensitive-data processing are unlikely to be more secure — is a testable hypothesis, not a result. The other concerns raised by the reader (abstract/Table 5 runtime mismatch, 'any asymmetric encryption' overstatement, and the 'no prior studies' novelty claim) are real but are presentation-level defects; they do not threaten the validity of the underlying measurements. The third-party classification heuristic and the upper-bound nature of the static analysis are secondary and generally make the paper's non-use estimate conservative. No machine-checked proof or independent replication is available, but the open-sourced dataset and tooling make the proposed sensitivity analysis feasible. A conditional verdict remains appropriate: the core measurements are valuable, but the headline adoption rates should be re-derived under an alternative denominator or explicitly bounded by label-error analysis.","tokens_in":26936,"tokens_out":8814,"duration_ms":87378,"concrete_test":"Perform a sensitivity analysis: rebuild the denominator using an independent proxy for sensitive-data processing, e.g., Play Store categories (Medical, Finance, Health & Fitness) plus apps requesting dangerous permissions (SMS, contacts, location, microphone) or, on a random sample of ~1,000 apps, dynamic traffic analysis to identify transmitted personal data. Recompute the 56.3% non-use and 5.03% StrongBox percentages under this alternative denominator and under a worst-case bound that treats all apps with missing data safety forms as sensitive. If the headline figures shift by more than 5 percentage points, the reliance on data safety labels is material and the abstract's generalization should be qualified; if they remain stable, the concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claims (56.3% non-use of trusted hardware; 5.03% StrongBox use) are ratios whose denominator is the set of apps 'self-reporting as processing sensitive user data' — 159,241 apps identified solely from Play Store data safety labels (§A.2). The paper itself notes (a) only 74.47% of apps have submitted a data safety form at crawl time, and (b) Google's definition of 'collected' covers data transmitted off-device, so an app that processes sensitive data locally but does not transmit it is excluded from the denominator. The paper offers the hypothesis that non-reporting apps are unlikely to use secure storage, but this is not evidence. If the excluded locally-processing apps differ in Keystore adoption from the labeled set — e.g., privacy-focused apps that store data locally could plausibly use the Keystore more often — then both the 56.3% and 5.03% figures are miscalibrated for the population the Discussion generalizes to ('apps collecting sensitive data'). The labels also permit both over- and under-reporting, and the paper does not bound the effect of label error on the headline ratios. This is load-bearing because the headline adoption statistics are the paper's primary contribution; the internal inconsistencies flagged by the reader (abstract vs Table 5, 'any asymmetric encryption', 'first study') are wording and novelty issues that do not change the measurement itself.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents KeyDroid, a static-analysis pipeline applied to 490,119 Android apps from AndroZoo, and uses it to measure adoption of the Android Keystore API (TEE-backed) and StrongBox (secure-element) across the ecosystem. Cross-referencing with Play Store data-safety labels, the authors report that 56.3% of apps self-reporting sensitive data do not reference trusted-hardware APIs and that 5.03% reference StrongBox. They further report that 94.7% of Keystore initialization calls reside in third-party libraries and that about 8.45% of hardware-backed keys disable randomized encryption. The paper also benchmarks symmetric and asymmetric operations across software, TEE, and StrongBox keystores on Pixel and other devices, finding TEE performance viable for most operations while StrongBox is slow for large payloads (e.g., 15.43 s for 1 MiB AES-GCM, 9.22 s for RSA-2048 key generation on Pixel 8). A small developer survey and a discussion of adoption barriers complete the paper.","tokens_in":27127,"tokens_out":6436,"duration_ms":64148,"significance":"If the measurements hold, this is the first systematic, large-scale study combining adoption and performance data for hardware-backed key storage on Android, and it provides useful empirical grounding for developer guidance. The open-sourced dataset and tooling (Section 11) are a valuable community resource, and the authors are appropriately careful to frame many usage statistics as upper bounds. The performance measurements are novel relative to prior work and directly address a gap in Android's documentation. However, the headline adoption percentages rest on self-reported Play Store data-safety labels and on static-analysis reachability heuristics, and the abstract and introduction contain internal numerical inconsistencies that will mislead readers if not corrected.","major_comments":[{"comment":"The denominator for the headline adoption figures (56.3% non-use, 5.03% StrongBox) is the set of 159,241 apps identified from Play Store data-safety labels. The paper itself acknowledges that only 74.47% of apps have submitted the form and that Google's definition of 'collected' covers data transmitted off-device, so apps that process sensitive data locally but do not transmit it are excluded from the denominator. The hypothesis in §A.2 that non-reporting apps are unlikely to use secure storage is not evidence. A sensitivity analysis against an independent sensitive-data proxy (e.g., permission-based classification or a manually labeled sample) is needed to bound the effect of label error on the headline ratios; without it, the Discussion's generalization to 'apps processing sensitive data' is not supported.","section":"§A.2, §4.1, §8"},{"comment":"The abstract's claim that StrongBox makes 'any kind of asymmetric encryption' infeasible is contradicted by the paper's own measurement in §5.1: asymmetric encryption of a 256-bit payload on the Pixel 8 takes 0.0125 s in StrongBox. The actual performance barrier is asymmetric key generation (9.22 s for RSA-2048) and large-payload operations, not small-payload asymmetric encryption. This wording should be corrected so the abstract accurately reflects the reported data.","section":"Abstract, §1, §5.1"},{"comment":"The Introduction states that encrypting a 1 MiB message with AES-GCM in StrongBox takes 'around 3 seconds' on the Pixel 8, but §5.1 and Table 5 report 15.43 s for this operation. This factor-of-five discrepancy is a factual error in the summary of the paper's own measurements and must be corrected before publication.","section":"§1 vs. §5.1 and Table 5"},{"comment":"The 56.3% non-use figure is presented in the abstract and Introduction as a precise measurement, but §4.1 defines the 43.7% usage rate as an upper bound and §7 states that the results provide an 'empirical upper bound' on secure key storage usage. Consequently, 56.3% is a lower bound on non-use, not an exact figure. The abstract should carry this qualifier (e.g., 'at least 56.3%' or 'as an upper bound, only 43.7%').","section":"§4.1, §7, Abstract"}],"minor_comments":[{"comment":"The reported runtimes are non-monotonic for both TEE (7.67 s at 8 MiB vs. 5.83 s at 10 MiB) and SE (159.61 s at 8 MiB vs. 127.01 s at 10 MiB), which is inconsistent with the claim that runtime increases linearly with message length; the entries should be checked or the measurement anomaly should be explained.","section":"§5.2, Table 5"},{"comment":"The developer survey received only 42 responses from 10,000 emails for the Sensitive-NonKeystore group and zero responses for the StrongBox-Disabled group; conclusions such as 'API usability did not appear to be a widespread concern' should be tempered given the low response rate, and the survey should not be used to support quantitative generalizations.","section":"§6"},{"comment":"The claims of being the 'first comprehensive survey' and 'first empirical analysis' should be hedged in light of the concurrently published work by Bove [33] on TEE-based API prevalence, which the paper itself cites; a phrase such as 'to the best of our knowledge' or a precise differentiator ('the first study combining usage and performance across TEE and SE') would be more accurate.","section":"Abstract, §1, §9"},{"comment":"The first/third-party classification heuristic (a package is third-party if referenced by multiple APKs from different developers) could misclassify first-party packages that are shared across apps from the same developer or libraries bundled under an app-specific package; a brief discussion of the expected error direction would help interpret the 94.7% third-party figure.","section":"§3.3, §4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's contribution is real and the dataset is valuable, but the abstract and introduction currently overstate the precision of the adoption figures, contain a direct numerical inconsistency (3 s vs. 15.43 s for StrongBox 1 MiB AES-GCM), and make an overly broad claim about asymmetric encryption that is contradicted by the paper's own measurements. The denominator sensitivity analysis for the data-safety labels is feasible and, in my view, necessary to support the headline claims. These are correctable within the manuscript's scope, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First, the useful part: this is the first large-scale empirical map of trusted hardware usage and performance on Android. The dataset is big (490k apps), the static analysis is careful (64 API methods, reachability, first-party vs third-party attribution), and the performance measurements across Pixel 2016–2023 plus other vendors are new and directly usable. StrongBox being roughly 35–55x slower than TEE for symmetric encryption, and 9.22s for RSA-2048 keygen, is concrete decision data that didn't exist in public form. The paper also ships its data and code, which puts a real floor under the results.\n\nThe adoption statistics (56.3% non-use, 5.03% StrongBox) are upper bounds by the paper's own admission, because static analysis can't prove reachability and the denominator comes from Play Store data-safety labels. The stress-test concern about the denominator is legitimate: the labels are self-reported, cover data transmitted off-device rather than processed locally, and only 74.47% of apps have submitted a form. So the denominator could be miscalibrated for the population the discussion generalizes to. That said, the paper clearly acknowledges this in §A.2 and frames the numbers as upper bounds, so this is a limitation to tighten rather than a fatal flaw. A sensitivity check against an independent proxy would be a meaningful revision but I don't think it overturns the main conclusion that trusted hardware adoption is low.\n\nThe bigger immediate problems are internal inconsistencies. The abstract says 1MiB AES-GCM takes \"around 3 seconds\" in StrongBox, but Table 5 says 15.43s on the Pixel 8. The abstract also says StrongBox makes \"any kind of asymmetric encryption\" infeasible, yet the paper's own measurement shows RSA-2048 encryption of a 256-bit payload takes 12.5ms. And the \"no prior studies\" claim in §2.2.2 is contradicted by the paper's own citations of Bove and Hugenroth et al. These are wording and framing errors, not measurement errors, but they matter for a headline-driven paper. The StrongBox adoption estimate also lacks uncertainty bounds.\n\nWho should read this: Android security researchers, platform security teams, and developers choosing between TEE and StrongBox. It deserves a serious referee. I'd send it to peer review with a request for a corrected abstract, a sensitivity analysis on the denominator, and clearer bounding language.","headline":"A large, mostly sound measurement study whose real contribution is the performance data; the adoption headlines are credible upper bounds but rest on a fragile label-based denominator and a few sloppy abstract claims.","tokens_in":27748,"tokens_out":2379,"would_cite":true,"duration_ms":26260,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Most Android apps handling sensitive data skip hardware-backed key storage; only 5.03% use the strongest option, StrongBox, which the paper finds too slow for common operations.","keywords":["hardware-backed key storage","Android Keystore","StrongBox secure element","trusted execution environment","Android app security","static analysis","Play Store data safety labels","cryptographic key performance"],"falsifier":"Re-run the paper's benchmark on a current flagship phone: if 1 MiB AES-GCM encryption in StrongBox completes in well under a second, or if RSA-2048 key generation finishes in under a second, the performance half of the claim is overturned. For the adoption half, instrument a random sample of the 159,241 apps labeled sensitive and check at runtime whether they actually store sensitive data on-device; if most never do, the 56.3% non-use figure is miscalibrated.","tokens_in":26657,"feed_emoji":"🔐","tokens_out":17146,"duration_ms":151594,"temperature":0.7,"pith_summary":"This paper asks whether Android apps actually use the hardware-backed key storage built into their devices, and whether performance explains the gaps. Analyzing 490,119 Play Store apps, the authors find that 56.3% of apps whose developers report collecting sensitive user data never reference the Android Keystore API at all, and only 5.03% use StrongBox, the strongest option, a secure element separate from the main processor. Their measurements show the trusted-execution-environment keystore is fast enough for nearly every common operation, while StrongBox takes 15.43 seconds to encrypt a single megabyte with AES-GCM, which rules it out for bulk symmetric encryption and for most asymmetric operations. Together the two halves quantify the gap between industry promotion of trusted hardware and actual developer uptake, and supply the performance numbers developers need to choose a security tier.","feed_headline":"56% of sensitive-data apps skip Android's hardware key vaults","feed_subtitle":"The strongest vault, StrongBox, takes 15 seconds to encrypt 1 MiB — the missing data behind low adoption.","key_machinery":"The central object is the Android Keystore API, which fronts two hardware backends: a trusted execution environment (a protected secure mode of the main processor, used by default) and StrongBox Keymaster, a secure element that must have its own CPU, tamper-resistant packaging, and a true random number generator, and that developers must explicitly request. The analysis instrument is KeyDroid, a static-analysis pipeline that decompiles each app, builds an inter-procedural call graph, extracts 64 Keystore-related API calls with their arguments, and classifies each call as first-party or third-party; the adoption claims ride on that classification, while the performance claims ride on a benchmarking app that times key generation, encryption, and signing across the software, TEE, and StrongBox keystores on physical devices. The thread joining the two halves is the trade-off the paper makes quantitative: stronger hardware costs runtime, and the measurements state exactly how much, per operation and per payload size.","core_discovery":"The central claim is that hardware-backed key storage is decisively underused across the Android ecosystem, and that the strongest tier, the secure element, is underused partly because it is impractically slow. On the usage side, of the 159,241 apps whose Play Store data safety labels report sensitive data collection, at most 43.7% reference the Keystore API, only 5.03% reference StrongBox, and 94.7% of key initialization calls sit inside third-party libraries rather than app-written code; the paper therefore presents its adoption figures as upper bounds on real secure storage. On the performance side, TEE-backed encryption is indistinguishable from software storage for payloads up to 1 MiB, whereas StrongBox on the Pixel 8 takes 15.43 seconds to encrypt 1 MiB with AES-GCM, 9.22 seconds to generate an RSA-2048 key, and over four minutes for 16 MiB, which the paper argues makes StrongBox infeasible for non-negligible payloads and most asymmetric operations.","pith_inferences":["Because data safety labels count only data transmitted off-device, apps that process sensitive data purely locally are invisible to the paper's denominator; if such apps are common, the true share of sensitive-data apps without hardware protection is larger than 56.3%, making the headline figures optimistic upper bounds.","The concentration of keystore calls in a handful of widely distributed SDKs suggests the cheaper adoption lever is library-level: app-store scanning and SDK vendors could push hardware-backed defaults where the keystore decision actually happens, and the paper's first-party-versus-third-party split makes that effect measurable within a release cycle.","The performance jump between secure-element hardware generations implies StrongBox usability is largely a property of the device fleet, not the app; a testable prediction is that StrongBox adoption tracks the installed base's age and that restricting analysis to devices containing secure elements would push the 5.03% figure upward."],"forward_implications":["Google's stated goal of making the secure element the 'lowest common denominator' for Android credential storage is far from met: only about one in ten apps that use any trusted hardware ever use StrongBox.","Developers can adopt TEE-backed keystores for typical payloads under 1 MiB with no user-visible performance penalty, so performance is not a valid reason to keep sensitive keys in a software keystore for those cases.","StrongBox is practical only for small payloads, such as wrapping a key that is later used in a faster keystore, or for one-time operations; the paper's tables let developers compute the exact cost of each choice.","Because 94.7% of key initialization calls come from third-party libraries, most trusted-hardware usage in the ecosystem is inherited from SDKs rather than chosen by the app's own developers.","The 8.45% of keystore-backed keys that disable randomized encryption show that secure API defaults do not by themselves guarantee secure deployments, since developers can and do deliberately opt out."],"supporting_citations":[{"why":"Supplies the large public corpus of Android app binaries from which the paper's dataset is drawn.","marker":"[6]"},{"why":"Defines the Android Keystore system, its TEE default, and the StrongBox option that the paper measures.","marker":"[24]"},{"why":"The data safety form is the sole basis for identifying apps that process sensitive data, the denominator of the adoption figures.","marker":"[75]"},{"why":"The bytecode analysis engine used to generate call graphs and extract the 64 tracked API calls.","marker":"[2]"},{"why":"Android's description of the secure element whose performance the paper benchmarks and whose qualitative 'a little slower' characterization it quantifies.","marker":"[11]"},{"why":"Prior estimate of secure-element availability across devices plus a small SE performance data point that this work extends into a comprehensive benchmark.","marker":"[50]"},{"why":"The earlier study of Android cryptographic misuse whose secure-configuration rules frame the paper's comparison of Keystore defaults with software APIs.","marker":"[42]"},{"why":"Earlier scan of key attestation usage that supplies the baseline for the paper's attestation adoption figure.","marker":"[51]"}],"fun_headline_variants":["Most sensitive Android apps skip hardware key vaults","Only 5% of Android apps use StrongBox key vault","StrongBox key vault too slow for many Android apps","Study: 56% of sensitive apps ignore Android key hardware","Android's secure key storage rarely used, performance costly"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that Play Store data safety labels, which are self-reported by developers and count data transmitted off-device rather than data processed locally, correctly identify which apps handle sensitive user data, because the entire denominator behind the 56.3% and 5.03% adoption figures is built from those labels.","fun_headline_variants_meta":{"raw":{"variants":["Most sensitive Android apps skip hardware key vaults","Only 5% of Android apps use StrongBox key vault","StrongBox key vault too slow for many Android apps","Study: 56% of sensitive apps ignore Android key hardware","Android's secure key storage rarely used, performance costly"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"total_tokens":1288,"prompt_tokens":1043,"completion_tokens":245,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":659,"completion_tokens_details":{"reasoning_tokens":166}},"tokens_in":659,"tokens_out":245,"duration_ms":3043,"temperature":1.0,"reasoning_tokens":166,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:29:09.378256+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the paper's benchmark on a current flagship phone: if 1 MiB AES-GCM encryption in StrongBox completes in well under a second, or if RSA-2048 key generation finishes in under a second, the performance half of the claim is overturned. For the adoption half, instrument a random sample of the 159,241 apps labeled sensitive and check at runtime whether they actually store sensitive data on-device; if most never do, the 56.3% non-use figure is miscalibrated.","supporting_citations":[{"cited_title":"Android Keystore system","cited_arxiv_id":null,"evidence_quote":"Defines the Android Keystore system, its TEE default, and the StrongBox option that the paper measures."},{"cited_title":"Provide information for Google Play’s Data safety section","cited_arxiv_id":null,"evidence_quote":"The data safety form is the sole basis for identifying apps that process sensitive data, the denominator of the adoption figures."},{"cited_title":"Sloth: Key stretching and deni- able encryption using secure elements on smartphones","cited_arxiv_id":null,"evidence_quote":"Prior estimate of secure-element availability across devices plus a small SE performance data point that this work extends into a comprehensive benchmark."},{"cited_title":"An empirical study of crypto- graphic misuse in android applications","cited_arxiv_id":null,"evidence_quote":"The earlier study of Android cryptographic misuse whose secure-configuration rules frame the paper's comparison of Keystore defaults with software APIs."},{"cited_title":"SARA: Secure Android Remote Authorization","cited_arxiv_id":null,"evidence_quote":"Earlier scan of key attestation usage that supplies the baseline for the paper's attestation adoption figure."}],"review_version":1}