{"id":"c3951a76-5be4-47cb-bf8e-a89701250265","arxiv_id":"2508.19493","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new 7,138-scenario benchmark shows mainstream smartphone AI agents frequently fail to notice or warn about private information.","lead":"This paper introduces SAPA-Bench, a 7,138-scenario test that measures whether smartphone AI agents recognize private information such as passwords and locations. In tests across seven agents, most missed a majority of privacy risks, and even the best scored only 67% on risk-aware responses.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"LLM judge for RA is unvalidated against humans and shares a generator with the benchmark; the exact RA scores and rankings remain unestablished.","rationale":"The central claim is that current smartphone agents have inadequate privacy awareness, evidenced by low PRR, PLAR, PCAR, and RA. The qualitative direction of this finding is likely robust, but the precise values and rankings depend crucially on the RA judge. The paper assigns RA using an LLM 'semantic alignment' with GPT-4o-generated reference prompts and never validates this judge against human raters, despite the claim that annotations passed human verification. The same LLM family generated the ground-truth responses and is also one of the evaluated models, so template-matching could inflate GPT-4o's RA relative to models with different phrasing conventions. This is the weakest point in the evaluation chain because it directly affects the headline result, the comparison of closed- versus open-source agents, and the 'best model' conclusion. The abstract-vs-table inconsistency on the 60% threshold further signals that the reported numbers should not be taken at face value. The reader identified this same concern, and the appropriate response is to keep the CONDITIONAL verdict: accept the qualitative conclusion only after the RA judge is validated and controls are added. I do not see evidence of intentional misrepresentation, and the human-verification pipeline is a genuine strength; the issue is methodological, not ethical. My recommendation is therefore UNCHANGED relative to the reader's CONDITIONAL verdict, with the concrete validation of the RA judge as the condition to be met.","tokens_in":12246,"tokens_out":2310,"duration_ms":23965,"concrete_test":"Take a stratified sample of 200 agent responses per RA condition (or 600 total), and have two independent human raters, blind to model identity and prompt condition, score each response for whether it warns about the privacy risk and appropriately refuses or requests confirmation. Compare the LLM judge against the majority human label; if Cohen's kappa is below 0.6, the judge is not reliably measuring RA. Also construct a control set of 200 non-sensitive instruction-screenshot pairs with neutral agent responses; if the LLM judge assigns nontrivial RA scores to neutral responses, the metric partly measures template overlap. Finally, re-run the judge with reference prompts removed or paraphrased; if RA scores shift materially, the metric depends on judge-side phrasing rather than agent behavior.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing step is the RA metric, whose definition depends on an LLM performing 'semantic alignment between the agent's response and a reference risk prompt' (Experiments, Evaluation Metrics). The reference risk prompts are generated by GPT-4o and human-verified, the judge is not validated against human raters, and GPT-4o is itself one of the benchmarked models. This creates a circular evaluation protocol: GPT-4o helped author the reference answers, an LLM judge rewards semantic similarity to those answers, and one of the models being scored is GPT-4o. A model that mimics GPT-4o's wording—or a judge that latches onto template phrases such as 'This action may result in...'—will receive inflated RA. The absence of non-sensitive control scenarios also means PRR cannot separate genuine detection from a disposition to flag everything, and RA has no neutral-response baseline. Compounding this, the abstract's claim that performance remains below 60% even with explicit hints is internally contradicted by Table 3, where Gemini 2.0-flash reaches 67.14 in the EH condition and the main text itself reports 67.14. The qualitative finding that open-source agents are weak at privacy awareness is plausible, but the reported RA magnitudes and the Gemini-vs-GPT-4o ranking are not established by the current evaluation protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces SAPA-Bench, a benchmark of 7,138 privacy-sensitive smartphone UI scenarios sampled from GUI-Odyssey and OS-Atlas, with annotations for privacy type, sensitivity level, leakage modality, and expected risk-warning responses. A five-stage pipeline combines GPT-4o-based automatic generation with human verification. Seven smartphone agents or generalist MLLMs are evaluated on five metrics (PRR, PLR, PLAR, PCAR, RA), and the paper reports that most agents have low privacy awareness, that open-source models trail closed-source models, that explicit hints improve risk awareness, and that higher-sensitivity scenarios are more detectable. The authors conclude that current smartphone agents lack adequate privacy safeguards and release code, data, and a project page.","tokens_in":12579,"tokens_out":7497,"duration_ms":64638,"significance":"If the findings hold, SAPA-Bench addresses a real gap: prior mobile-agent benchmarks measure task competence, not privacy awareness. The paper's strengths include the large scale (7,138 scenarios), the use of a two-pass human verification pipeline for ground-truth labels, the multi-dimensional metric design, and the public release of code and data. The qualitative direction of the findings—especially the weakness of open-source agents—is plausible and is supported by PRR, PLR, PLAR, and PCAR even if RA is set aside. However, the exact RA magnitudes and the Gemini-versus-GPT-4o ranking rest on an unvalidated LLM judge that shares a generator with the benchmark, and the PRR metric is computed without negative controls; these issues must be fixed before the quantitative claims can be relied upon.","major_comments":[{"comment":"The abstract's headline claim that performance \"remains below 60% even with explicit hints\" is directly contradicted by Table 3, where Gemini 2.0-flash reaches 67.14% in the Explicit Hint condition, and by the main text's statement that \"Gemini to 67.14%.\" The abstract should be revised to say \"almost all models\" or otherwise exempt Gemini, and the exact prompting condition should be stated.","section":"Abstract; Results and Discussion, Table 3"},{"comment":"The RA metric relies on an LLM performing semantic alignment between the agent's response and a reference risk prompt, but those reference prompts were generated by GPT-4o in Stage 1, GPT-4o is itself one of the seven benchmarked models, and no human validation of the LLM judge is reported. Because Stage 2 forces ground-truth responses into a standardized \"This action may result in ... Please confirm before proceeding\" template, the judge may reward template-like phrasing rather than genuine privacy awareness. The exact RA scores and the Gemini-over-GPT-4o ranking are therefore not established by the current protocol; the authors should validate the judge against human ratings, report inter-rater agreement, and include reference-free or neutral-response baselines.","section":"Experiments, Evaluation Metrics; Annotation Pipeline, Stage 2"},{"comment":"PRR is defined as the proportion of all samples that the agent flags as privacy-related, and every one of the 7,138 benchmark scenarios is privacy-sensitive by construction; there are no non-sensitive control scenarios. As a result, a model that always says \"this is private\" would receive 100% PRR, so the reported low PRR values cannot by themselves support the claim that \"the vast majority of sensitive scenarios go undetected.\" To separate detection from over-flagging, the benchmark needs a set of non-sensitive or neutral scenarios and a false-positive or over-flagging statistic.","section":"Experiments, Evaluation Metrics (PRR); Dataset Statistics"}],"minor_comments":[{"comment":"\"Evaluation Matrics\" is a typo; it should be \"Evaluation Metrics.\"","section":"Experiments, Evaluation Metrics"},{"comment":"Model names are inconsistent across the paper: \"LLaV A-NeXT\" appears alongside \"LLaVA-NeXT,\" and \"InternVL 1.5\" appears in one place while \"InternVL2.5\" is used elsewhere; please standardize.","section":"Table 1; Experiments"},{"comment":"The column header layout for PLR (Image/Instruction/Overall) is ambiguous, and it is not immediately clear that the RA column in Table 1 corresponds to the Explicit Hint condition only; please clarify the header and state the condition explicitly.","section":"Table 1"},{"comment":"The introduction says performance remains \"below 60% even with implicit hints,\" while the abstract says \"explicit hints\"; these statements should be aligned, and the Gemini value should be corrected in both places.","section":"Introduction; Abstract"},{"comment":"The text reports \"GPT-4o and Gemini score 55.03% and 66.14%, respectively,\" but Table 3 lists 67.14% for Gemini under the Explicit Hint condition; the inconsistency should be corrected.","section":"Results and Discussion"},{"comment":"The paper states that two annotators independently review each sample but reports no inter-annotator agreement statistic; adding Cohen's kappa or a similar measure for the privacy type, sensitivity level, and location labels would strengthen the claim of high-quality ground truth.","section":"Annotation Pipeline, Stage 4"},{"comment":"No confidence intervals or standard errors are reported for any of the five metrics; with 7,138 scenarios, bootstrap or analytic intervals would clarify which model differences are meaningful.","section":"Results and Discussion; Tables 1-3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially valuable, but the current RA metric and the abstract's quantitative claim need substantial work. My main concern is that the exact numerical rankings may be artifacts of the LLM judge and the absence of negative controls; a human-validated judge and non-sensitive control scenarios should settle this. The abstract inconsistency is straightforward to fix but must not be overlooked."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: they built a large, human-verified benchmark for privacy awareness in smartphone agents, and the finding that today's agents are weak at this is likely correct. But the exact RA scores and the abstract's 'below 60%' claim are not supported by the paper as written, so treat the rankings with caution.\n\nWhat's genuinely new: 7,138 scenarios sampled from real GUI corpora, eight privacy categories, three sensitivity levels, per-scenario annotations for exposure modality, leakage location, expected warning, and a five-metric evaluation protocol. The two-pass human verification on every sample is a real strength and rare in this kind of benchmark. I also think the qualitative ordering—closed-source models clearly better than open-source, sensitivity level matters, explicit hints help—is robust enough to survive fixes to the metrics.\n\nWhere it gets soft. First, PRR is computed only on privacy-positive scenarios. Since every scenario in the benchmark is privacy-sensitive, PRR is just the true positive rate. A model that warns on every input would score 100%. Without non-sensitive control scenarios, you cannot interpret PRR as 'recognition' rather than a tendency to warn. That is a genuine gap, not a nitpick.\n\nSecond, the RA metric. They use an LLM to compare the agent's response against a reference risk prompt. The reference prompts were generated by GPT-4o, and GPT-4o is one of the seven benchmarked models. The judge has no reported human validation. So the specific RA values—and in particular the Gemini vs GPT-4o comparison—are not established. It's plausible that the judge rewards overlap with GPT-4o's phrasing. This needs a human-validated judge on a sample, plus a neutral-response baseline, before I'd trust the magnitudes.\n\nThird, the abstract states RA stays 'below 60% even with explicit hints,' but Table 3 shows Gemini at 67.14% under explicit hints and the abstract itself says 'RA of 67%.' That's a direct internal contradiction and needs fixing. There are also small typos (Gemini 66.14 vs 67.14 in the text; InternVL 1.5 vs 2.5 naming).\n\nMinor: no inter-annotator agreement is reported despite the two-pass review; adding that would strengthen the annotation quality argument.\n\nThe central message is plausible and probably survives revision. The paper is worth a serious referee, but it needs major revision on evaluation validity before the specific numbers are citable. For anyone working on mobile agent safety or privacy evaluation, this is a useful starting point.","headline":"A substantial human-checked privacy benchmark for phone agents, but the RA judge and PRR denominator need work before the exact scores can be trusted.","tokens_in":13011,"tokens_out":3418,"would_cite":true,"duration_ms":31217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 7,138-scenario benchmark shows smartphone agents fail to warn users about most privacy-sensitive operations.","keywords":["SAPA-Bench","privacy awareness","smartphone agents","multimodal large language models","privacy leakage","risk awareness","GUI agents","privacy-sensitive scenarios"],"falsifier":"Re-score RA on a random subset using human judges who are blind to the reference risk prompts, then compare the resulting ranking with the paper's LLM-judge ranking; if models that score high under the LLM judge fall behind or vice versa, the RA metric is measuring protocol artifacts rather than privacy awareness. Alternatively, run a live deployment probe in which an agent is given a task like pasting a password and the actual behavior is observed; if an agent with high RA still executes without warning, the benchmark's claim about privacy awareness fails to predict behavior.","tokens_in":12022,"feed_emoji":"🔒","tokens_out":5242,"duration_ms":44388,"temperature":0.7,"pith_summary":"Contemporary smartphone agents that act on screen content are granted extensive access to personal data, and this paper asks whether they recognize and warn about privacy-sensitive operations before executing them. It introduces SAPA-Bench, a benchmark of 7,138 real-world scenarios sampled from popular apps, each annotated with privacy type, sensitivity level, exposure modality, and an expected warning response. It then defines five metrics—privacy recognition, localization, level awareness, category awareness, and risk awareness—and evaluates seven off-the-shelf agents and vision-language models. The central finding is that privacy awareness is markedly inadequate across the board: even the best-performing closed-source model reaches a risk-awareness score of only 67.14%, while open-source models score around 30% or lower on recognition and near-zero on granular classification. The paper concludes that dedicated privacy training, alignment, and evaluation are urgently needed, and that prompt hints can partly compensate but do not solve the problem.","feed_headline":"Smartphone agents miss most privacy risks, benchmark shows","feed_subtitle":"Across 7,138 scenarios, even the best model warns about privacy only 67% of the time","key_machinery":"The benchmark's core object is SAPA-Bench, a dataset of 7,138 privacy-sensitive scenarios created by filtering roughly 80,000 screenshots from GUI Odyssey and OS-Atlas, auto-generating instruction-response pairs with GPT-4o, then verifying them through a two-pass human review. Each scenario carries annotations for privacy presence, exposure modality (screenshot or instruction), one of eight privacy categories, a three-tier sensitivity level, and an expected warning response. Five metrics—PRR, PLR, PLAR, PCAR, and RA—measure recognition, localization, severity estimation, category classification, and risk-aware response generation respectively. The RA metric uses an LLM judge to compare agent responses against reference risk prompts, while the other four compare against human-verified labels.","core_discovery":"Every tested agent fails to reliably detect, localize, classify, or warn about privacy leakage in smartphone tasks. Across 7,138 scenarios, all models score below 85% on Privacy Recognition Rate; open-source systems such as Show-UI, SpiritSight Agent, Qwen2.5-VL, and InternVL2.5 sit near 30%, meaning most sensitive scenarios go unnoticed. Fine-grained metrics are worse: privacy level awareness and category awareness hover in the single to low-double digits for most models, and risk awareness peaks at 67.14% even for the best model under explicit hints. Higher-sensitivity scenarios are more likely to be recognized than low- and medium-sensitivity ones, and closed-source models generally outperform open-source ones across all metrics. The paper interprets this as evidence that current agents optimize task completion without the privacy guardrails needed for safe deployment.","pith_inferences":["The RA metric's LLM judge compares semantic alignment with a GPT-4o-written reference risk prompt, so scores may partly reward response style or template overlap rather than a genuine decision to halt or warn; re-scoring with human raters or a judge blind to the reference wording would test this.","The benchmark's instruction-response pairs are also GPT-4o-generated, so the dataset may inherit GPT-4o's privacy biases; a fully human-authored subset or a perturbation test with paraphrased instructions would reveal whether results change.","A natural extension is a live-environment probe: deploy an agent on a real device, inject a PIN-entry or clipboard-paste task, and measure whether the agent actually pauses and warns before acting, comparing against SAPA-Bench's predicted RA.","The finding that higher-sensitivity scenarios are easier to detect suggests a possible training curriculum: ordering synthetic privacy training data by sensitivity level could give open-source models a cheap alignment signal."],"forward_implications":["Agents currently lack the basic prerequisite for safe autonomy: they do not reliably know when an operation touches credentials, contacts, payment, or location.","Even the strongest commercial model falls short of deployment-grade privacy reliability, so off-the-shelf agents should not be entrusted with sensitive operations without an external privacy-check module.","Explicit prompt hints reliably improve risk awareness across models, so prompt-level interventions are a low-cost partial mitigation while training-based fixes are developed.","Low- and medium-sensitivity privacy events are the ones most often missed, meaning the largest privacy risk is not dramatic credential theft but ordinary contextual leakage such as pasted emails or shared meeting links.","The open-source and closed-source gap indicates that alignment data and post-training, not just base vision-language ability, drive privacy behavior; reproducing that alignment in open models is a concrete next target."],"supporting_citations":[{"why":"Supplies the GUI Odyssey screenshot corpus that the benchmark filters and samples for privacy-sensitive scenarios.","marker":"(Lu et al. 2024)"},{"why":"Supplies OS-Atlas screenshots as the other source of real-world UI scenarios in the benchmark.","marker":"(Wu et al. 2024b)"},{"why":"MobileSafetyBench is the prior safety benchmark the paper positions against, motivating the full privacy perception pipeline rather than benign-versus-harmful task classification.","marker":"(Lee et al. 2024)"},{"why":"GPT-4o is used for automatic instruction-response generation and is one of the evaluated closed-source models.","marker":"(Hurst et al. 2024)"},{"why":"Cited by the paper as the Gemini 2.0-flash reference; Gemini is the best-performing closed-source agent evaluated.","marker":"(Reid et al. 2024)"},{"why":"LLaVA-NeXT is one of the open-source models whose stronger privacy perception is attributed to harmful-content filtering and chain-of-thought training.","marker":"(Liu et al. 2024)"},{"why":"Qwen2.5-VL technical report is used to discuss the baseline gap between strong OCR and VQA performance and weak privacy detection.","marker":"(Bai et al. 2025a)"}],"fun_headline_variants":["Almost all phone AI agents score below 60% privacy awareness","Best phone AI agent detects privacy risks only 67% of time","Privacy blind spot: phone AI agents miss majority of risks","Phone agents flunk privacy test: even top model at 67%","AI phone agents have weak privacy awareness, benchmark shows"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's measurements of privacy awareness assume that the GPT-4o-generated, human-verified ground truth and the LLM-based risk-awareness judge capture genuine privacy understanding rather than rewarding wording similar to the reference prompts.","fun_headline_variants_meta":{"raw":{"variants":["Almost all phone AI agents score below 60% privacy awareness","Best phone AI agent detects privacy risks only 67% of time","Privacy blind spot: phone AI agents miss majority of risks","Phone agents flunk privacy test: even top model at 67%","AI phone agents have weak privacy awareness, benchmark shows"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2464,"prompt_tokens":966,"completion_tokens":1498,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":1412}},"tokens_in":582,"tokens_out":1498,"duration_ms":14322,"temperature":1.0,"reasoning_tokens":1412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:51:37.029384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score RA on a random subset using human judges who are blind to the reference risk prompts, then compare the resulting ranking with the paper's LLM-judge ranking; if models that score high under the LLM judge fall behind or vice versa, the RA metric is measuring protocol artifacts rather than privacy awareness. Alternatively, run a live deployment probe in which an agent is given a task like pasting a password and the actual behavior is observed; if an agent with high RA still executes without warning, the benchmark's claim about privacy awareness fails to predict behavior.","supporting_citations":[],"review_version":2}