{"id":"aab95cd9-c67d-4045-825e-0a8d64bb6487","arxiv_id":"2608.09049","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"TaxoScale scales taxonomy construction to 601,257 privacy/security app-review pseudo-labels, reporting higher path, level, coverage, and novelty scores than four baselines evaluated on a 100K subset.","lead":"TaxoScale extends an expert-built privacy and security taxonomy to more than 600,000 mobile app reviews using recursive hierarchical clustering and LLM-based naming. The paper reports that the pipeline beats four automatic-taxonomy baselines on path, level, coverage, and novelty metrics and releases the filtered corpus and code.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison is confounded: baselines run on a 100K subset while TaxoScale runs on 601K pseudo-labels; coverage and novelty gains may reflect data volume rather than algorithmic superiority.","rationale":"The reader's strongest concern is exactly the one I identify: the comparison in Table 1 is between TaxoScale on 601,257 pseudo-labels and baselines on a 100K subset, so the headline outperformance claim is confounded by data volume. This is not an internal inconsistency or a methodological fraud; it is an experimental design gap that is explicitly disclosed in Appendix E. The coverage metric, in particular, is structurally biased toward the method that saw the full corpus, because it measures coverage over the full 601K pseudo-label set. Novelty is also likely inflated for the full-data method, since the subsample may miss rare but novel themes. The other concerns raised by the reader—GPT-4o as judge and hyperparameter selection on novelty—are secondary; the matched-data issue alone is sufficient to justify the conditional verdict. I do not propose moving to reject: the pipeline, corpus release, and qualitative novel branches are valuable, and a matched-data rerun could plausibly restore the claim. Thus the appropriate disposition remains conditional acceptance pending the matched-data evaluation.","tokens_in":11670,"tokens_out":3234,"duration_ms":31752,"concrete_test":"Run TaxoScale on the exact 100K representative subset used for the baselines (k-means with k=1,000 and 100 pseudo-labels per cluster), keeping θ, δleaf, ε, and the naming prompts fixed. Then compute all four metrics over the same 601,257 pseudo-labels on which the original TaxoScale is evaluated. If TaxoScale's coverage drops from 0.9852 toward the baselines' levels (e.g., near 0.9090) and its novelty drops toward 0.4219, the reported margins are largely explained by data volume, and the comparison should be re-run on equal footing. If TaxoScale on 100K still outperforms all baselines on the full pseudo-label set, the confound is not decisive.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Appendix E states that Chain-of-Layers, TaxoAdapt, and SCYCHIC are given 'a representative 100K-pseudo-label subset' while TaxoScale uses all 601,257 pseudo-labels. Coverage is measured as the fraction of the full 601,257 pseudo-labels whose maximum cosine similarity to any leaf exceeds 0.6. A taxonomy built from only 100K pseudo-labels cannot plausibly cover themes that appear only in the omitted 501K labels, so TaxoScale's coverage margin (0.9852 vs. 0.9090) may be a direct artifact of seeing six times more data. Weighted novelty is similarly affected: novel nodes discovered in the full corpus are weighted by pseudo-label count and may simply not exist in a 100K subsample. The paper treats scale as both the motivation and the claimed achievement, but the 'outperforms baselines' claim cannot be separated from the data-volume confound without a matched-data evaluation. Because the central conclusion rests on Table 1, this unequal comparison is the most load-bearing weakness in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TaxoScale, a pipeline for automatic taxonomy construction from mobile app reviews at scale. The authors aggregate 18.63M reviews, filter them using an LLM classifier (Llama-3.3-70B) to obtain 601,257 privacy/security-related pseudo-labels, and then extend an expert-built seed taxonomy via recursive hierarchical clustering with k-means and Ward linkage, binary-to-multiway conversion, and LLM-based node naming. They report that TaxoScale outperforms four baselines (Chain-of-Layers, TaxoCom, TaxoAdapt, SCYCHIC) on path, level, coverage, and novelty metrics on the 601K corpus, and they release the filtered corpus, pseudo-labels, and code.","tokens_in":12025,"tokens_out":4642,"duration_ms":41625,"significance":"If the claimed results hold, the contribution is significant: it is among the first to push hierarchy induction to the hundreds-of-thousands scale, and it releases a privacy/security app review resource that the community currently lacks. The seed-anchored approach and the recursive clustering scheme are sensible, and the runtime analysis for Ward linkage is useful. However, the comparison is confounded by unequal data volumes, the novelty metric is tuned on the evaluated quantity, and the coverage metric is a self-consistent reconstruction score. These issues must be addressed before the central claim can be accepted.","major_comments":[{"comment":"The headline comparison in Table 1 is confounded by data volume: Appendix E states that Chain-of-Layers, TaxoAdapt, and SCYCHIC are run on a representative 100K-pseudo-label subset, while TaxoScale uses all 601,257 pseudo-labels. Coverage and novelty are both corpus-dependent: a taxonomy built from a 100K sample cannot cover themes that occur only in the omitted 501K labels, so the reported margins (e.g., Coverage 0.9852 vs 0.9090) may reflect data volume rather than algorithmic quality. Please run the baselines on the full set (or TaxoScale on the same 100K subset), and report the metrics on a common subsample at minimum.","section":"Section 4.3, Table 1, Appendix E"},{"comment":"The height-tolerance epsilon is selected by sweeping the same weighted novelty metric that is later reported as the headline Novelty score in Table 1: the text states that 'weighted novelty peaks at epsilon=20 and drops on either side,' and this value is then used to produce the final taxonomy. This hyperparameter selection makes the reported novelty advantage (0.4778 vs 0.4219) partly a tuning artifact. Provide a sensitivity table for all four metrics over a range of epsilon and delta_leaf, and, if possible, evaluate with held-out pseudo-labels or a validation criterion that is independent of the metric being reported.","section":"Appendix E (epsilon selection)"},{"comment":"The Coverage score measures the fraction of pseudo-labels whose maximum cosine similarity to any leaf exceeds 0.6, using the same Qwen3 embeddings that were used to construct the leaves in the first place. This is a self-reconstruction score: it is expected to be high by construction and provides limited evidence of semantic taxonomy quality. The claim that coverage shows TaxoScale's ability to 'exploit large-scale pseudo-labels' is not supported unless compared against a random-leaf baseline, a flat k-means baseline, or human-rated coverage on a sample.","section":"Section 4.2, Table 4"},{"comment":"Path, Level, and Novelty scores all depend on GPT-4o as the judge, yet the paper provides no validation that GPT-4o's judgments agree with human judgments on app-review taxonomy structures. Because these metrics are the basis for the claimed superiority over baselines, a small human-annotated sample (e.g., 50-100 parent-child pairs and nodes) with reported inter-annotator agreement (e.g., Cohen's kappa) between GPT-4o and humans would substantially strengthen the evaluation.","section":"Section 4.2, Section 4.3"}],"minor_comments":[{"comment":"The heading 'Algortihm' is a typo for 'Algorithm'.","section":"Appendix C heading"},{"comment":"Figure 2 is referenced in Section 4.4 but placed in Appendix F; a cross-reference is fine, but the reader would benefit from an explicit note in the main text that the figure appears there.","section":"Section 4.4 / Appendix F"},{"comment":"Table 5 reports runtime and memory for the Ward component only; it would be useful to report the end-to-end wall-clock time for the full TaxoScale pipeline on 601,257 pseudo-labels.","section":"Table 5 / Section 4.1"},{"comment":"The description of the 100K subset ('k-means clustering (k=1,000) and sampling 100 pseudo-labels per cluster') should specify the random seed and the distance metric used for k-means so that the subset construction is reproducible.","section":"Appendix E"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely problem and the release of the filtered corpus, pseudo-labels, and code is a valuable community resource. However, the central empirical claim in the abstract and Table 1 is not yet supported because of the data-volume confound and the tuning of epsilon on the novelty metric itself. A matched-data comparison and sensitivity analysis are essential. I do not see a fundamental algorithmic flaw, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. The recursive clustering method is genuinely new, and the 601K-review corpus is a real resource. The evaluation, however, doesn't support the headline claim as written: baselines get 100K pseudo-labels, TaxoScale gets all 601K, and the main margins are likely an artifact of data volume.\n\nWhat's actually new: the algorithm alternates top-down mini-batch k-means with bottom-up Ward merging on centroids, recursing on subclusters above a tractability threshold. That is a sensible way to push hierarchical clustering to hundreds of thousands of points, and the runtime table justifies the threshold. They also release the filtered reviews, extracted spans, and code. The classifier and span-extractor development is careful: ten LLMs times eight prompt templates, with human-annotated benchmarks and kappa 0.87 and 0.78.\n\nWhere the soft spots are, in proportion. The comparison in Table 1 is unequal. Appendix E says Chain-of-Layers, TaxoAdapt, and SCYCHIC run on a 100K subset sampled by k-means. Coverage is the fraction of all 601K pseudo-labels within 0.6 cosine similarity of a leaf. A taxonomy built from 100K points cannot be expected to cover themes that appear only in the other 501K, so the coverage margin (0.985 vs 0.909) mostly measures data volume, not algorithmic superiority. Weighted novelty has the same confound, and epsilon was set by sweeping on weighted novelty—that is tuning on the test metric. The path and level scores come from GPT-4o with no human agreement check. Individually these are addressable; together they mean the central claim is not yet established.\n\nWhat holds up: the high-level approach is sound, the runtime scaling is real, and the discovered branches look plausible. The paper is also honest: the 100K subset is disclosed, and the limitations section is reasonable.\n\nWho this is for: anyone building taxonomies from large review corpora, and privacy/security requirements analysts. The corpus and code are the most immediately useful part.\n\nRecommendation: send to peer review. The right referee should ask for a matched-data evaluation—either run TaxoScale on 100K or scale the baselines up—and validate the LLM judgments on a sample of nodes against human raters. With those changes this becomes a solid methods paper; as it stands it is a strong preliminary report.","headline":"A useful scaling trick and a big new corpus, but the headline win over baselines is confounded: baselines see 100K pseudo-labels, TaxoScale sees 601K, so the claimed margins aren't trustworthy yet.","tokens_in":12443,"tokens_out":4255,"would_cite":true,"duration_ms":36776,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"TaxoScale extends an expert privacy taxonomy to 601,257 app reviews and beats four automatic-taxonomy baselines on path, level, coverage, and novelty.","keywords":["taxonomy generation","mobile app reviews","privacy and security","recursive hierarchical clustering","large language models","pseudo-label filtering","coverage and novelty metrics"],"falsifier":"Run TaxoAdapt, SCYCHIC, Chain-of-Layers, and TaxoCom on the full 601,257 pseudo-labels with the same embedding model and compute, then re-measure the four metrics; if any baseline reaches TaxoScale's path, level, coverage, or novelty scores, the claim that TaxoScale's machinery is responsible for the gains fails.","tokens_in":11475,"feed_emoji":"🔐","tokens_out":6541,"duration_ms":53738,"temperature":0.7,"pith_summary":"The paper tries to show that taxonomies of privacy and security concerns in mobile app reviews need not be hand-crafted and static. It introduces TaxoScale, a four-step pipeline that seeds from an existing expert taxonomy and extends it with embedding-based recursive clustering and LLM-based node naming, applied to 601,257 reviews filtered from 18.63 million. TaxoScale is reported to outperform four automatic-taxonomy baselines on all four metrics, and to surface branches, such as network security and behavioral tracking, that prior taxonomies omit. The practical stake is that privacy and security concerns in app stores change as apps and permission models evolve, and an automatically updatable taxonomy could track those shifts at corpus scale.","feed_headline":"TaxoScale beats four baselines on 601K-review privacy taxonomy","feed_subtitle":"Clustering plus LLM naming extends an expert taxonomy and surfaces novel branches such as network security.","key_machinery":"The load-bearing mechanism is Recursive Hierarchical Clustering, an algorithm that alternates top-down mini-batch k-means partitioning with bottom-up Ward agglomerative merging on cluster centroids, recursing on any subcluster larger than a tractability threshold and otherwise applying Ward linkage directly to raw embeddings. This yields a binary tree; height-tolerance collapsing then merges near-equal-height splits into multiway nodes, and Qwen3-235B names each new node from its pseudo-labels, parent path, and sibling names. The k-means/Ward alternation is what lets clustering scale beyond a few thousand documents while preserving fine-grained leaf coherence.","core_discovery":"On the paper's own terms, the central discovery is that a hybrid clustering pipeline can turn a hand-built seed taxonomy into a much larger, still-coherent taxonomy of 601,257 concern pseudo-labels. Recursive Hierarchical Clustering alternates top-down k-means partitioning with bottom-up Ward merging so that agglomerative clustering, normally quadratic and intractable at this size, is applied only to subclusters of at most 40,000 points. After binary-to-multiway conversion and LLM naming, TaxoScale attains path 0.7285, level 0.7726, coverage 0.9852, and novelty 0.4778, beating TaxoAdapt, SCYCHIC, Chain-of-Layers, and TaxoCom on those metrics and adding new branches including IP address management, VPN functionality, and financial transaction tracking.","pith_inferences":["Because baselines see only a 100,000-item subset while TaxoScale sees all 601,257 items, an equal-data comparison would isolate whether the reported advantage comes from the recursion or from sheer data volume.","The novelty and path scores ultimately rest on GPT-4o's judgments; a human-annotated gold standard on a sample of nodes would show whether those judgments are transferable.","The alternating k-means/Ward recursion may transfer beyond taxonomies, for example to large-scale topic detection in streaming text, since it attacks the same quadratic bottleneck."],"forward_implications":["Updates can be triggered automatically as new reviews accumulate, so privacy and security categories do not have to be frozen at design time.","The pipeline itself is domain-agnostic, so the same four steps could be applied to any large corpus; the paper leaves that cross-domain validation to future work.","At the 0.6 similarity threshold, TaxoScale covers 98.52 percent of pseudo-labels, meaning almost every filtered concern can be attached to a named leaf.","The newly surfaced branches give privacy researchers concrete starting points for manual inspection and further annotation."],"supporting_citations":[{"why":"Supplies the expert seed taxonomy that TaxoScale extends.","marker":"Akgul et al. (2024)"},{"why":"TaxoAdapt is the strongest baseline on path and level, combining LLM knowledge with corpus signals.","marker":"Kargupta et al. (2025)"},{"why":"SCYCHIC is the best baseline on novelty, alternating top-down embedding clustering with bottom-up abstraction.","marker":"Gao et al. (2025)"},{"why":"Chain-of-Layers is the baseline with the best coverage among competitors, prompting an LLM top-down.","marker":"Zeng et al. (2024)"},{"why":"TaxoCom is the baseline that expands a seed hierarchy by discovering novel topic clusters.","marker":"Lee et al. (2022)"},{"why":"Qwen3-Embedding-8B is the encoder that produces the pseudo-label and descriptor embeddings.","marker":"Zhang et al. (2025b)"},{"why":"Qwen3-235B-A22B-2507 is the LLM that names taxonomy nodes.","marker":"Yang et al. (2025)"}],"fun_headline_variants":["Clustering + LLM builds 600K-review privacy taxonomy","TaxoScale: scalable privacy taxonomy from 601K app reviews","LLM-named clusters expand privacy taxonomy to 601K reviews","From 600K app reviews, TaxoScale mines new privacy branches","Hybrid pipeline scales privacy taxonomy to 601K reviews"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes a baseline run on a 100,000-pseudo-label subset is a fair stand-in for how that baseline would perform on all 601,257 labels, so part of the reported advantage might be data volume rather than the algorithm.","fun_headline_variants_meta":{"raw":{"variants":["Clustering + LLM builds 600K-review privacy taxonomy","TaxoScale: scalable privacy taxonomy from 601K app reviews","LLM-named clusters expand privacy taxonomy to 601K reviews","From 600K app reviews, TaxoScale mines new privacy branches","Hybrid pipeline scales privacy taxonomy to 601K reviews"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000741,"raw_usage":{"total_tokens":3275,"prompt_tokens":880,"completion_tokens":2395,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":496,"completion_tokens_details":{"reasoning_tokens":2316}},"tokens_in":496,"tokens_out":2395,"duration_ms":15991,"temperature":1.0,"reasoning_tokens":2316,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:16:13.249597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TaxoAdapt, SCYCHIC, Chain-of-Layers, and TaxoCom on the full 601,257 pseudo-labels with the same embedding model and compute, then re-measure the four metrics; if any baseline reaches TaxoScale's path, level, coverage, or novelty scores, the claim that TaxoScale's machinery is responsible for the gains fails.","supporting_citations":[],"review_version":1}