Pith. sign in

REVIEW 4 major objections 4 minor 19 references

Security and Privacy Taxonomy Generation from Mobile App Reviews

T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TaxoScale extends an expert privacy taxonomy to 601,257 app reviews and beats four automatic-taxonomy baselines on path, level, coverage, and novelty.

desk verdict A useful scaling trick and a big new corpus, but the headline win over baselines is confounded: baselines see 100K pseudo-labels, TaxoScale sees 601K, so the claimed margins aren't trustworthy yet. read the letter →

arxiv 2608.09049 v1 pith:YMOLVVSO submitted 2026-08-10 cs.CL

Moghis Fereidouni , Vinaik Chhetri , Umar Farooq , A.B. Siddique This is my paper · ORCID
classification cs.CL
keywords taxonomygenerationmobileappreviewsprivacyandsecurityrecursivehierarchicalclusteringlargelanguagemodelspseudo-labelfilteringcoveragenoveltymetrics
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that taxonomies of privacy and security concerns in mobile app reviews need not be hand-crafted and static. It introduces TaxoScale, a four-step pipeline that seeds from an existing expert taxonomy and extends it with embedding-based recursive clustering and LLM-based node naming, applied to 601,257 reviews filtered from 18.63 million. TaxoScale is reported to outperform four automatic-taxonomy baselines on all four metrics, and to surface branches, such as network security and behavioral tracking, that prior taxonomies omit. The practical stake is that privacy and security concerns in app stores change as apps and permission models evolve, and an automatically updatable taxonomy could track those shifts at corpus scale.

What carries the argument

The load-bearing mechanism is Recursive Hierarchical Clustering, an algorithm that alternates top-down mini-batch k-means partitioning with bottom-up Ward agglomerative merging on cluster centroids, recursing on any subcluster larger than a tractability threshold and otherwise applying Ward linkage directly to raw embeddings. This yields a binary tree; height-tolerance collapsing then merges near-equal-height splits into multiway nodes, and Qwen3-235B names each new node from its pseudo-labels, parent path, and sibling names. The k-means/Ward alternation is what lets clustering scale beyond a few thousand documents while preserving fine-grained leaf coherence.

What would settle it

Run TaxoAdapt, SCYCHIC, Chain-of-Layers, and TaxoCom on the full 601,257 pseudo-labels with the same embedding model and compute, then re-measure the four metrics; if any baseline reaches TaxoScale's path, level, coverage, or novelty scores, the claim that TaxoScale's machinery is responsible for the gains fails.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a hybrid clustering pipeline can turn a hand-built seed taxonomy into a much larger, still-coherent taxonomy of 601,257 concern pseudo-labels. Recursive Hierarchical Clustering alternates top-down k-means partitioning with bottom-up Ward merging so that agglomerative clustering, normally quadratic and intractable at this size, is applied only to subclusters of at most 40,000 points. After binary-to-multiway conversion and LLM naming, TaxoScale attains path 0.7285, level 0.7726, coverage 0.9852, and novelty 0.4778, beating TaxoAdapt, SCYCHIC, Chain-of-Layers, and TaxoCom on those metrics and adding new branches including IP address management, VPN functionality, and financial transaction tracking.

Load-bearing premise

The comparison assumes a baseline run on a 100,000-pseudo-label subset is a fair stand-in for how that baseline would perform on all 601,257 labels, so part of the reported advantage might be data volume rather than the algorithm.

Editorial extensions

If this is right

  • Updates can be triggered automatically as new reviews accumulate, so privacy and security categories do not have to be frozen at design time.
  • The pipeline itself is domain-agnostic, so the same four steps could be applied to any large corpus; the paper leaves that cross-domain validation to future work.
  • At the 0.6 similarity threshold, TaxoScale covers 98.52 percent of pseudo-labels, meaning almost every filtered concern can be attached to a named leaf.
  • The newly surfaced branches give privacy researchers concrete starting points for manual inspection and further annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because baselines see only a 100,000-item subset while TaxoScale sees all 601,257 items, an equal-data comparison would isolate whether the reported advantage comes from the recursion or from sheer data volume.
  • The novelty and path scores ultimately rest on GPT-4o's judgments; a human-annotated gold standard on a sample of nodes would show whether those judgments are transferable.
  • The alternating k-means/Ward recursion may transfer beyond taxonomies, for example to large-scale topic detection in streaming text, since it attacks the same quadratic bottleneck.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents TaxoScale, a pipeline for automatic taxonomy construction from mobile app reviews at scale. The authors aggregate 18.63M reviews, filter them using an LLM classifier (Llama-3.3-70B) to obtain 601,257 privacy/security-related pseudo-labels, and then extend an expert-built seed taxonomy via recursive hierarchical clustering with k-means and Ward linkage, binary-to-multiway conversion, and LLM-based node naming. They report that TaxoScale outperforms four baselines (Chain-of-Layers, TaxoCom, TaxoAdapt, SCYCHIC) on path, level, coverage, and novelty metrics on the 601K corpus, and they release the filtered corpus, pseudo-labels, and code.

Significance. If the claimed results hold, the contribution is significant: it is among the first to push hierarchy induction to the hundreds-of-thousands scale, and it releases a privacy/security app review resource that the community currently lacks. The seed-anchored approach and the recursive clustering scheme are sensible, and the runtime analysis for Ward linkage is useful. However, the comparison is confounded by unequal data volumes, the novelty metric is tuned on the evaluated quantity, and the coverage metric is a self-consistent reconstruction score. These issues must be addressed before the central claim can be accepted.

major comments (4)
  1. [Section 4.3, Table 1, Appendix E] The headline comparison in Table 1 is confounded by data volume: Appendix E states that Chain-of-Layers, TaxoAdapt, and SCYCHIC are run on a representative 100K-pseudo-label subset, while TaxoScale uses all 601,257 pseudo-labels. Coverage and novelty are both corpus-dependent: a taxonomy built from a 100K sample cannot cover themes that occur only in the omitted 501K labels, so the reported margins (e.g., Coverage 0.9852 vs 0.9090) may reflect data volume rather than algorithmic quality. Please run the baselines on the full set (or TaxoScale on the same 100K subset), and report the metrics on a common subsample at minimum.
  2. [Appendix E (epsilon selection)] The height-tolerance epsilon is selected by sweeping the same weighted novelty metric that is later reported as the headline Novelty score in Table 1: the text states that 'weighted novelty peaks at epsilon=20 and drops on either side,' and this value is then used to produce the final taxonomy. This hyperparameter selection makes the reported novelty advantage (0.4778 vs 0.4219) partly a tuning artifact. Provide a sensitivity table for all four metrics over a range of epsilon and delta_leaf, and, if possible, evaluate with held-out pseudo-labels or a validation criterion that is independent of the metric being reported.
  3. [Section 4.2, Table 4] The Coverage score measures the fraction of pseudo-labels whose maximum cosine similarity to any leaf exceeds 0.6, using the same Qwen3 embeddings that were used to construct the leaves in the first place. This is a self-reconstruction score: it is expected to be high by construction and provides limited evidence of semantic taxonomy quality. The claim that coverage shows TaxoScale's ability to 'exploit large-scale pseudo-labels' is not supported unless compared against a random-leaf baseline, a flat k-means baseline, or human-rated coverage on a sample.
  4. [Section 4.2, Section 4.3] Path, Level, and Novelty scores all depend on GPT-4o as the judge, yet the paper provides no validation that GPT-4o's judgments agree with human judgments on app-review taxonomy structures. Because these metrics are the basis for the claimed superiority over baselines, a small human-annotated sample (e.g., 50-100 parent-child pairs and nodes) with reported inter-annotator agreement (e.g., Cohen's kappa) between GPT-4o and humans would substantially strengthen the evaluation.
minor comments (4)
  1. [Appendix C heading] The heading 'Algortihm' is a typo for 'Algorithm'.
  2. [Section 4.4 / Appendix F] Figure 2 is referenced in Section 4.4 but placed in Appendix F; a cross-reference is fine, but the reader would benefit from an explicit note in the main text that the figure appears there.
  3. [Table 5 / Section 4.1] Table 5 reports runtime and memory for the Ward component only; it would be useful to report the end-to-end wall-clock time for the full TaxoScale pipeline on 601,257 pseudo-labels.
  4. [Appendix E] The description of the 100K subset ('k-means clustering (k=1,000) and sampling 100 pseudo-labels per cluster') should specify the random seed and the distance metric used for k-means so that the subset construction is reproducible.

Circularity Check

2 steps flagged · score 6.0 of 10

Partial circularity: epsilon is tuned on the reported novelty metric, and coverage measures reconstruction of the same pseudo-labels used to build the leaves.

  1. fitted input called prediction [Appendix E (epsilon sweep), Section 4.2 (Weighted Novelty metric), Table 1]
    "In our sweep, weighted novelty peaks at ϵ= 20 and drops on either side... We therefore set ϵ= 20."

    Weighted Novelty is reported in Table 1 as evidence of TaxoScale's superiority (0.4778 vs 0.4219 for SCYCHIC). Appendix E discloses that the hyperparameter epsilon was selected by sweeping this same metric on the corpus: 'weighted novelty peaks at ϵ=20... We therefore set ϵ=20.' The reported novelty is therefore a fitted maximum of the target metric rather than an independent predictive result. Baselines are not given an equivalent target-metric tuning, so the novelty comparison is partially forced by the choice of epsilon.

  2. self definitional [Section 4.2 (Coverage Score), Sections 3.1-3.3 (assignment, clustering, leaf extraction), Appendix E (100K vs 601K data volumes)]
    "Coverage Score: The fraction of pseudo-labels whose maximum cosine similarity to any leaf node exceeds 0.6, indicating how well leaves account for corpus topics."

    The pseudo-labels evaluated for coverage are exactly the items used to construct the taxonomy: Section 3.1 assigns each pseudo-label to a node, Section 3.2 clusters these pseudo-label embeddings, and Section 3.3 extracts leaves from those clusters. Coverage then asks whether each pseudo-label lies within 0.6 cosine similarity of any leaf. A leaf built from a cluster will naturally be close to its own member pseudo-labels, so high coverage is in part a self-reconstruction score rather than an independent measure of how well leaves account for external topics. The comparison is further confounded by Appendix E's unequal data volumes: baselines receive a representative 100K-pseudo-label subset while TaxoScale uses all 601,257 pseudo-labels.

full rationale

The paper's central empirical claim is not entirely reducible: path and level scores are judged by GPT-4o on parent-child specificity and sibling coherence, and these judgments are not fitted by construction or imported from the authors' own prior work. The seed taxonomy of Akgul et al. (2024) is external, and the baselines are independent methods. However, two of the four headline metrics are partially forced. Coverage is defined over the same pseudo-labels used to build the leaves, so it measures in large part how well clusters reconstruct their own inputs; the exact leaf-vector construction is not specified, which leaves some room for a non-tautological reading, but the metric is nonetheless self-referential in definition. Weighted novelty is explicitly used to tune epsilon, so the reported novelty score is an optimized value of the target metric rather than an independent estimate. The unequal data-volume setup (100K for baselines vs 601,257 for TaxoScale) is a serious confound for the coverage and novelty comparisons, but it is better classified as a correctness risk than as circularity. No load-bearing self-citation chain was found; the data-source citations to prior work by the same group are not used to justify the taxonomy-construction claim. On balance, the headline superiority claim is partially circular because two of its four supporting metrics reduce, at least in part, to the pipeline's own construction choices and fitting procedure.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or conceptual entities are proposed. The paper introduces a pipeline name, not an entity. Free parameters are mostly conventional clustering thresholds, but delta_leaf and epsilon are tuned on the reported metrics.

free parameters (5)
  • theta (assignment threshold) = 0.5
    Threshold for attaching pseudo-labels to seed taxonomy nodes in Section 3.1; no sensitivity analysis or derivation is provided.
  • tau (tractability threshold) = 40,000
    Chosen from runtime and memory measurements in Table 5 so direct Ward linkage remains feasible.
  • delta_leaf (leaf height threshold) = 10
    Chosen to balance leaf coherence vs redundant siblings; described as a sweep in Appendix E.
  • epsilon (height-tolerance for collapsing) = 20
    Appendix E states weighted novelty peaks at epsilon=20 and drops on either side; the value is selected on the same metric later reported as a result.
  • Kmax (k-means cap)
    In Algorithm 1 the cluster count uses kmax but the numerical value is not stated in the paper.
assumptions (4)
  • domain assumption Cosine similarity between Qwen3 embeddings and seed-node descriptors reflects semantic relevance to privacy/security categories.
    The whole assignment step in Section 3.1 uses theta=0.5 on cosine similarity as the attachment criterion; no validation that this threshold maps to semantic correctness.
  • domain assumption The seed taxonomy of Akgul et al. (2024) is a correct and complete structuring of privacy/security concerns.
    TaxoScale seeds every branch from this 20-category taxonomy; if it is outdated or biased, 'novel branches' are just subdivisions of existing categories.
  • domain assumption GPT-4o judge scores for path, level, and novelty are valid and unbiased measures of taxonomy quality.
    The evaluation metrics in Section 4.2 entirely rely on one LLM judge; no human agreement study or judge calibration is reported.
  • domain assumption The 300-review and 140-review human benchmarks are sufficient to establish that the classifier and extractor generalize to 18.63M and 601K reviews.
    Filter and summarizer are deployed at full scale based on F1 0.97 (n=300) and Jaccard 0.85 (n=140).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Security and Privacy Taxonomy Generation from Mobile App Reviews." pith.science (2026). https://pith.science/paper/YMOLVVSO

@misc{pith2026260809049,
  author       = {Pith},
  title        = {Pith review of: Security and Privacy Taxonomy Generation from Mobile App Reviews},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YMOLVVSO}},
  note         = {Machine review of arXiv:2608.09049}
}
read the original abstract

Mobile app reviews are a rich, continuously renewing source of how users experience privacy and security, yet existing taxonomies of these concerns are hand-crafted and cannot keep pace with the evolving nature of the data. Automating taxonomy construction is the natural response, but scalability is the core challenge: current LLM- and clustering-based methods are developed for scientific corpora of a few thousand documents and do not extend to app review collections numbering in the hundreds of thousands. We address this gap in two ways. First, we filter app reviews for privacy- and security-related content, yielding a comprehensive corpus of over 600K reviews. Second, we introduce TaxoScale, a pipeline that handles taxonomy construction at this scale by extending an expert-defined taxonomy via Recursive Hierarchical Clustering and LLM-based node naming. TaxoScale outperforms strong automatic-taxonomy baselines on path, level, coverage, and novelty metrics, and discovers novel branches absent from prior taxonomies.

Figures

Figures reproduced from arXiv: 2608.09049 by the authors.

Figure 1
Figure 1. Our four-step taxonomy construction pipeline: (1) embedding and pseudo-label assignment, (2) recursive [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Taxonomy branches for tracking, password, network security, and parental control. [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 9 canonical work pages

  1. [3]

    Hamza Harkous, Sai Teja Peddinti, Rishabh Khandel- wal, Animesh Srivastava, and Nina Taft

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948. Hamza Harkous, Sai Teja Peddinti, Rishabh Khandel- wal, Animesh Srivastava, and Nina Taft

    arXiv

  2. [4]

    InFindings of the Association for Computational Linguistics: ACL 2024, pages 118–132, Bangkok, Thailand

    CHIME: LLM-assisted hierarchical organization of scientific studies for liter- ature review support. InFindings of the Association for Computational Linguistics: ACL 2024, pages 118–132, Bangkok, Thailand. Association for Com- putational Linguistics. Yuntong Hu, Zhuofeng Li, Zheng Zhang, Chen Ling, Raasikh Kanjiani, Boxin Zhao, and Liang Zhao

    work page 2024

  3. [5]

    Albert Q

    Taxonomy tree generation from citation graph.arXiv preprint arXiv:2410.03761. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Men- sch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guil- laume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thoma...

    arXiv

  4. [7]

    Mixtral of experts.arXiv preprint arXiv:2401.04088. Joe H. Ward Jr

    arXiv

  5. [8]

    InFind- ings of the Association for Computational Linguistics: EMNLP 2024, pages 8838–8855, Miami, Florida, USA

    Knowl- edge navigator: LLM-guided browsing framework for exploratory search in scientific literature. InFind- ings of the Association for Computational Linguistics: EMNLP 2024, pages 8838–8855, Miami, Florida, USA. Association for Computational Linguistics. Timoteo Kelly, Abdulkadir Korkmaz, Samuel Mallet, Connor Souders, Sadra Aliakbarpour, and Praveen Rao

    work page 2024

  6. [9]

    arXiv preprint arXiv:2506.19268

    Health app reviews for privacy & trust (harpt): A corpus for analyzing patient privacy con- cerns, trust in providers and trust in applications. arXiv preprint arXiv:2506.19268. Dongha Lee, Jiaming Shen, Seongku Kang, Susik Yoon, Jiawei Han, and Hwanjo Yu

    work page arXiv

  7. [11]

    InProceedings of the 46th International ACM SI- GIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 3007–3016, New York, NY , USA

    Mobilerec: A large scale dataset for mobile apps recommendation. InProceedings of the 46th International ACM SI- GIR Conference on Research and Development in Information Retrieval, SIGIR ’23, page 3007–3016, New York, NY , USA. Association for Computing Machinery. K. V . Mardia, J. T. Kent, and J. M. Bibby. 1979.Mul- tivariate Analysis. Probability and M...

    work page 1979

  8. [14]

    Preprint, arXiv:2503.19786

    Gemma 3 technical report. Preprint, arXiv:2503.19786. Mengting Wan, Tara Safavi, Sujay Kumar Jauhar, Yujin Kim, Scott Counts, Jennifer Neville, Siddharth Suri, Chirag Shah, Ryen W. White, Longqi Yang, Reid Andersen, Georg Buscher, Dhruv Joshi, and Nagu Rangan

    arXiv

Show all 19 references
  1. [15]

    InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 10626–10649, Singapore

    Goal-driven explainable clustering via language de- scriptions. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Process- ing, pages 10626–10649, Singapore. Association for Computational Linguistics. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang,...

    2023

  2. [16]

    Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Zhenwen Liang, Zhihan Zhang, and Meng Jiang

    Qwen3 technical report.arXiv preprint arXiv:2505.09388. Qingkai Zeng, Yuyang Bai, Zhaoxuan Tan, Shangbin Feng, Zhenwen Liang, Zhihan Zhang, and Meng Jiang

    arXiv

  3. [18]

    InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15616–15634, Suzhou, China

    Context-aware hierar- chical taxonomy generation for scientific papers via LLM-guided multi-aspect clustering. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15616–15634, Suzhou, China. Association for Computational Lin- guistic...

    2025

  4. [2011]

    Preksha Nema, Pauline Anthonysamy, Nina Taft, and Sai Teja Peddinti

    Meth- ods of hierarchical clustering.arXiv preprint arXiv:1105.0121. Preksha Nema, Pauline Anthonysamy, Nina Taft, and Sai Teja Peddinti

    arXiv

  5. [2018]

    InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 2701–2709, New York, NY , USA

    Taxogen: Unsupervised topic taxonomy construction by adaptive term embedding and clus- tering. InProceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’18, page 2701–2709, New York, NY , USA. Association for Computing Machinery. ...

    arXiv

  6. [2020]

    InPro- ceedings of The Web Conference 2020, WWW ’20, page 1908–1919, New York, NY , USA

    Nettaxo: Automated topic tax- onomy construction from text-rich network. InPro- ceedings of The Web Conference 2020, WWW ’20, page 1908–1919, New York, NY , USA. Association for Computing Machinery. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ram...

    2020

  7. [2022]

    InProceedings of the ACM Web Conference 2022, WWW ’22, page 2819–2829, New York, NY , USA

    Taxocom: Topic taxonomy completion with hierarchical discovery of novel topic clusters. InProceedings of the ACM Web Conference 2022, WWW ’22, page 2819–2829, New York, NY , USA. Association for Computing Machinery. M. H. Maqbool, Umar Farooq, Adib Mosharrof, A. B. Siddique, a...

    2022

  8. [2023]

    Mistral 7b.Preprint, arXiv:2310.06825. Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bam- ford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 oth- ers

    arXiv

  9. [2024]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, and 1 others

    arXiv

  10. [2025]

    arXiv preprint arXiv:2504.13834

    Science hierarchogra- phy: Hierarchical organization of science literature. arXiv preprint arXiv:2504.13834. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others

  11. [4096]

    tracks my location

    E Implementation Details Chain-of-Layers, TaxoAdapt, and SCYCHIC are limited by their LLMs’ context windows, so we provide them a representative 100K-pseudo-label subset obtained by k-means clustering (k=1,000) and sampling 100 pseudo-labels per cluster. For the Recursive Hier...

    2011

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.