{"id":"26cb8c20-9280-4eb4-98d0-57340df103f6","arxiv_id":"2509.04091","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A new 6,055-app Android benchmark with version-level third-party library labels shows ten existing detection tools top out at 60% F1, with many far lower, exposing R8-era fragility and weak version resolution.","lead":"This paper builds TPL-Benchmark, a dataset of 6,055 Android apps with version-level third-party library labels, and uses it to test ten detection tools. The tools' best library-level F1 is 60.15%, and several are far worse under modern R8 optimizations, showing current detection trails real-world apps.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth validation shows only 50% exact triplet match; version-level labels and all tool F1 scores in Table 4 are therefore not yet established.","rationale":"The paper's strongest claim is that TPL-Benchmark is reliable at scale and that current tools perform poorly, with the best library-level F1 at 60.15%. This claim depends on the ground-truth labels actually matching the APK contents, especially at version granularity. The paper's own validation shows only 50% exact triplet match with Gradle's resolved dependency set. The correlation and MAE metrics are count-based and cannot reveal whether differences are missing/extra libraries or version misassignments. Since version-level annotation is a core contribution, this is a genuine soft spot. I do not see a reason to reject the paper: the evaluation protocol is careful, the manual validation is described in detail, and the dataset is promised for release. However, acceptance should remain conditional until the artifacts and a mismatch breakdown are independently verifiable. The proposed concrete test would settle the concern by quantifying the mismatch types and their impact on the reported F1 scores.","tokens_in":25156,"tokens_out":4476,"duration_ms":47816,"concrete_test":"On a fresh stratified sample of n=200 from the released dataset, independently execute Gradle's releaseRuntimeClasspath, compare Lp and Lgold triplet-by-triplet, and classify each mismatch as missing, extra, or version-only. Then recompute the version-level F1 of the best candidate-set tool (LibPecker) and the highest-recall tool (LibHunter) using the corrected labels on the affected apps. If version-only mismatches exceed 5% of labeled triplets, or either F1 shifts by more than 3 points, the version-level benchmark and the headline performance ranking are not robust.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.4.1 reports r=0.99, MAE=0.75, and exact-match rate=0.50 between TPL-Extractor's Lp and Gradle releaseRuntimeClasspath Lgold on n=200. The central claim—that TPL-Benchmark provides trustworthy version-level labels at scale—rests on this validation. Exact match at 50% means half the sampled projects have at least one triplet wrong, and MAE/count correlation cannot detect version mismatches because a wrong version leaves the count unchanged. The residual errors are attributed to build-configuration-induced and plugin-resolved dependencies, but no breakdown is given, nor is version-only disagreement reported. Since Eq. (7) approximates Gradle resolution rather than executing it, version conflicts, BOMs, dynamic versions, and local imports are resolved heuristically; Appendix H concedes the result is a practical approximation rather than an absolute standard. If a nontrivial share of the 50% mismatches are wrong versions, every version-level tool score in Table 4 and the downstream CVE mappings shift, and the headline 'highest F1=60.15%' is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents TPL-Benchmark, a dataset of 6,055 Android apps with version-level third-party library (TPL) annotations in <Group ID, Artifact ID, Version> form, built from APK–source pairs harvested from F-Droid and GitHub. The labels are produced by a new tool, TPL-Extractor, which parses Gradle build files, resolves variables, BOMs, and version conflicts, and aggregates dependencies from reachable modules. The authors report an internal reliability check on 200 apps against Gradle's releaseRuntimeClasspath resolution (r=0.99, MAE=0.75, exact-match rate 0.50), and then use the dataset to evaluate ten TPL detection tools, obtaining library-level F1 scores up to 60.15% and version-level F1 scores much lower. They also use the dataset for downstream analyses: CVE mapping, malware feature-dilution analysis, secret leakage scanning, and LLM-based code evaluation. The paper argues that current TPL detection tools are far from practical deployment quality and that TPL-Benchmark provides a more realistic and larger-scale evaluation foundation than prior datasets.","tokens_in":25474,"tokens_out":2964,"duration_ms":30676,"significance":"If the ground-truth labels are trustworthy, the dataset is a substantial contribution: it is far larger than existing open TPL benchmarks, contains version-level annotations, spans DX/D8/R8-era builds, and includes local and remote dependencies. The unified evaluation of ten tools under one protocol is also valuable, and the downstream analyses illustrate concrete uses of such a benchmark. The paper's open-science commitment is appropriate and necessary for the claims. However, the validity of every headline number—tool F1 scores, CVE mapping counts, and downstream statistics—depends on the correctness of the version-level labels. The current internal validation is not sufficient to establish that correctness, so the paper's central contribution is not yet fully supported.","major_comments":[{"comment":"The benchmark's central claim is trustworthy version-level labels, but the validation on n=200 reports only a 50% exact-match rate between Lp and Gradle's releaseRuntimeClasspath. MAE=0.75 and r=0.99 are count-based metrics and cannot detect version mismatches: replacing one version with another leaves the count unchanged. The paper attributes residual errors to build-configuration-induced and plugin-resolved dependencies but gives no breakdown and does not report version-only disagreement. Since Eq. (7) approximates Gradle's resolution rather than executing it, errors from BOM expansion, dynamic versions, and local imports may be shared by both Lp and Lgold. Appendix H concedes that the ground truth is 'a practical approximation rather than an absolute standard.' I request a version-level error breakdown, an independent validation against packaged binaries (e.g., comparing classes/resou","section":"§3.4.1, Eq. (11)"},{"comment":"The evaluation protocol for candidate-set tools gives them a strong advantage that is not reflected in the abstract or Table 4. The candidate set includes the ground-truth libraries themselves, plus 100 random other versions and 100 random non-used libraries. In real deployment, a detection tool would not receive its target library as a candidate. This makes the reported F1=60.15% for LibPecker an upper-bound-like estimate under a favorable condition, not a realistic operating point. The full-database tools are evaluated under a different, more difficult setting. The paper should state this asymmetry prominently and analyze how results change when candidate sets are constructed without ground-truth knowledge.","section":"§4.1 and Appendix G"},{"comment":"The data preparation step excludes 841 ground-truth jars 'with little or no code features' and selects 946 apps 'mostly with >20 TPLs.' The exact selection criterion and the distribution of excluded apps are not reported. If apps with few TPLs or small libraries are systematically excluded, the benchmark's difficulty is biased toward library-heavy apps, which may inflate or deflate tool scores in ways that change the generalizability of the conclusions. Please report the full selection rule, the number of apps excluded at each step, and the sensitivity of Table 4 to these exclusions.","section":"§4.1"}],"minor_comments":[{"comment":"The column header 'Paris' in Table 2 appears to be a typo; presumably it should be 'Pairs.' Please correct and align the table formatting.","section":"Table 2"},{"comment":"The figure contains the typo 'Retreval'; it should read 'Retrieval.' Also, the figure is dense and hard to read in monochrome; consider separating the pipeline stages.","section":"Figure 1"},{"comment":"The LLM evaluation uses 'Qwen-Plus model [56]' but the cited reference is the Qwen2 technical report. Please clarify the exact model version and API access date, and report the prompt template or provide it in an appendix for reproducibility.","section":"§5.4"},{"comment":"The '100 randomly selected' candidate libraries are not reproducible without a seed or a released list. Since candidate selection affects F1, please release the candidate sets and seeds.","section":"Appendix G"},{"comment":"The malware analysis uses a 'curated TPL whitelist' that is not described in the paper. Please explain how this whitelist was derived from TPL-Benchmark and whether it includes version information.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The main risk is not malicious overclaiming but insufficient validation for the load-bearing ground-truth assumption. The dataset artifacts and validation details are promised only 'upon acceptance,' which makes it difficult for reviewers to verify the central claim now. I would encourage the editor to require the validation data and code for the 200-app check as part of the revision, rather than deferring all artifacts to camera-ready. If the authors cannot supply the version-level breakdown or independent binary-based validation, the paper's headline conclusions should be substantially softened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading despite its central weakness. The real contribution is the dataset construction pipeline: TPL-Extractor handles Gradle variable resolution, BOM expansion, module reachability, and version conflict resolution in a systematic way, and the scale is beyond anything open in the field — 6,055 real apps, 15,274 versions, 62,218 mappings. The unified evaluation of ten tools under one protocol is also genuinely useful, and the finding that most tools fall apart on R8-era apps is important and probably correct.\n\nThe soft spot is exactly where the stress-test note points. Validation on 200 apps shows r=0.99 and MAE=0.75, but exact triplet match is only 50%. That means half the projects have at least one wrong triplet. The paper attributes the residual to build-configuration and plugin-resolved dependencies, but it doesn't say how many are wrong versions versus extra or missing libraries. Version-level tool scores and the CVE mappings depend on versions, so this matters. Also, the golden truth is Gradle's own dependency resolution, so errors shared by the extractor and Gradle won't be caught. The paper honestly calls the ground truth a \"practical approximation\" in Appendix H, but the intro and abstract advertise \"precise version-level annotations.\" That tension needs to be resolved.\n\nOther issues are secondary. The candidate-set protocol for some tools (ground truth plus 100 random decoys) is not fully realistic, even if it's a reasonable engineering compromise. Excluding 841 jars with little code could bias results. And the dataset isn't public yet, so the central artifact can't be inspected.\n\nThese are not fatal. The headline result — that current tools are far from practical accuracy — would survive even with some label noise, because the performance gaps are enormous. But the exact F1 scores and the version-level claims are conditional on the ground-truth quality. The paper deserves a serious referee, and the referees should push for a breakdown of the mismatch types, release of the validation artifacts, and a clearer description of how the candidate sets were chosen. I'd accept it for peer review with major revision required.","headline":"A serious and useful empirical study of TPL detection, but the version-level ground truth is not yet firmly established: the 50% exact-match rate on the validation sample is a load-bearing caveat that needs a breakdown and public artifacts.","tokens_in":25910,"tokens_out":2443,"would_cite":false,"duration_ms":24702,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 6,055-app, version-level ground truth shows the best Android third-party library detector reaches only 60.15% F1, and the worst 2.26%.","keywords":["third-party library detection","Android security","ground truth benchmark","software composition analysis","version-level detection","R8 obfuscation resilience","supply chain vulnerabilities","secret leakage"],"falsifier":"Compile a fresh, stratified sample of TPL-Benchmark projects spanning DX-era and R8 full-mode AGP versions and compare TPL-Extractor's triplets against Gradle's resolved release-runtime classpath: if exact-match falls materially below the reported 50%, or version mismatches concentrate in modern builds, the version-level trustworthiness claim that anchors every tool score is undermined. A second check: run a non-code-feature detector (resources plus build metadata) on the same 946-app evaluation; if it clears roughly 85% library-level F1, the paper's attribution of failure to code-level method","tokens_in":25121,"feed_emoji":"🧩","tokens_out":9634,"duration_ms":79654,"temperature":0.7,"pith_summary":"This paper argues that the field of Android third-party library (TPL) detection has been evaluating itself on small, unrepresentative datasets, and builds the largest open-source benchmark yet—6,055 real apps with version-level library labels—to test that premise. The central finding is that ten published detectors, run under a unified protocol, all fall far short of deployment quality: the best library-level F1 is 60.15%, the worst 2.26%, and version-level accuracy is weaker still. The paper traces the failures to R8-era code shrinking, namespace ambiguity between libraries sharing a root package, a systematic blind spot for Kotlin-only libraries, and feature sparsity in optimized code. It then shows why the gap matters: over a quarter of the apps carry at least one CVE-affected library, stripping libraries removes 60–80% of benign code without hurting malware detection accuracy, and 829,580 secrets leak across 19.2% of projects. If the benchmark is right, every downstream tool that assumes reliable library identification inherits an unmeasured margin of error.","feed_headline":"Best Android library detector hits just 60% F1 on 6,055 apps","feed_subtitle":"A version-level ground truth shows supply-chain, vulnerability, and malware analysis all inherit the detection gap.","key_machinery":"The load-bearing object is the version-level triplet <Group ID, Artifact ID, Version>, produced by the TPL-Extractor operator E(p) = ρ(∆(N(D, A, K))): N normalizes every dependency declaration in reachable modules into explicit triplets (resolving variables, TOML catalogs, BOM imports, and interpolated strings), ∆ removes exact duplicates, and ρ resolves version conflicts the way Gradle does. The same triplet serves as ground truth for scoring detectors and as the join key for mapping libraries to CVEs, apps to CVEs, and call-graph nodes to libraries. The benchmark's distributional coverage—R8-era builds, legacy DX, local .jar/.aar/.so imports, and a wide version spread—is what lets the pape","core_discovery":"The paper's claim is that TPL-Benchmark—6,055 APK/source pairs annotated with <Group ID, Artifact ID, Version> triplets, covering 5,756 unique libraries, 15,274 versions, and 62,218 app-library links—is a trustworthy ground truth at scale, and that under it the ten state-of-the-art TPL detectors it evaluates are far from practical usefulness. On 946 apps with 28,210 valid library links, the best library-level F1 was 60.15% (LibPecker) and the worst 2.26% (Orlis); version-level F1 topped out near 49% (LibID with candidate sets), and only one tool (LibHunter) kept recall above 76%. The labels themselves come from TPL-Extractor, which parses Gradle build files into normalized triplets, deduplic","pith_inferences":["If adopted as a standard, the benchmark would likely compress the reported performance spread: the 2–60% F1 range is partly an artifact of each tool being tuned on its own dataset, and a common benchmark would re-anchor thresholds and candidate-selection heuristics across the board.","The 50% exact-match validation rate invites a decomposition the paper does not provide: version errors versus missing or extra libraries have different consequences, since version errors poison CVE-to-app mappings while missing libraries shrink recall baselines.","A natural next experiment the paper implies but does not run: a detector built on resource files plus build metadata, tested on this same benchmark, should beat every code-only tool—the root-cause analysis predicts this directly.","The secret-leakage channel analysis suggests build artifacts (.snapshot, .dill, .rsp) are an under-scanned vector; scanning CI caches and generated outputs rather than source files alone would be a concrete policy takeaway beyond the paper's stated recommendations."],"forward_implications":["A detector that reaches deployment quality on this benchmark cannot rely on code-structural signatures alone; the root-cause analysis points to compiler-aware matching and non-code artifacts such as resources and build metadata that survive R8 transformations.","Version-level accuracy is the binding constraint: with the best version F1 near 49%, any vulnerability report built on current detectors understates which apps run a vulnerable version of a library.","Library stripping is a cheap win for analysis pipelines: removing up to 70% of call-graph nodes held malware detection F1 at 86–88% while cutting storage from 238 GB to 71 GB and feature extraction time by 20%.","Software composition analysis should surface not just 'CVE present' but version lag and patch-availability-without-adoption, since nearly 40% of apps with 10+ CVEs target modern AGP versions.","The Kotlin blind spot—no detector identifies libraries that only contain kotlin_metadata—will widen as Kotlin Multiplatform distribution grows."],"supporting_citations":[{"why":"LibScout defines the foundational similarity-based signature approach that the paper evaluates and finds fragile under R8-era shrinking.","marker":"[23]"},{"why":"LibLOOM's two-stage bloom-filter detector posts the best efficiency in the evaluation, setting the speed/accuracy balance reference.","marker":"[36]"},{"why":"LibPecker achieves the best library-level F1 (60.15%), setting the ceiling the paper's central claim hinges on.","marker":"[78]"},{"why":"LibScan's candidate-set version-level detection supplies the comparison point for version discrimination accuracy.","marker":"[65]"},{"why":"LibID's two variants set the version-level F1 reference (~41–49%) and demonstrate the prohibitive runtime of precise matching.","marker":"[77]"},{"why":"LibHunter, the compiler-aware detector with the highest recall, is the paper's key evidence that optimization-aware methods help.","marker":"[68]"},{"why":"ORLIS, an obfuscation-resilient detector, posts the worst F1 (2.26%), anchoring the lower bound of the performance spread.","marker":"[61]"},{"why":"LibRadar represents the clustering-based category, showing fast but coarse package-structure output is too imprecise for reliable use.","marker":"[48]"},{"why":"LibD is the other clustering baseline, evaluated for scalability but limited to ambiguous obfuscated package structures.","marker":"[46]"},{"why":"AndroZoo supplies the benign and malicious app corpus used for the malware downstream study after TPL stripping.","marker":"[19]"}],"fun_headline_variants":["New ground truth shows 10 TPL tools struggle: best F1 60%, version-level 49%","Supply chain blind spot: TPL detection tops out at 60% F1 on 6,055 apps","Android library detection gap: 10 tools tested, best F1 60% on 6,055 apps","R8-era apps break TPL tools: benchmark reveals 60% F1 ceiling"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The labels extracted from Gradle build files correspond to what is actually packaged inside the paired APK, including the correct version; on the paper's own 200-app validation, only 50% of label sets matched Gradle's independently resolved dependency set exactly.","fun_headline_variants_meta":{"raw":{"variants":["New ground truth shows 10 TPL tools struggle: best F1 60%, version-level 49%","Supply chain blind spot: TPL detection tops out at 60% F1 on 6,055 apps","Android library detection gap: 10 tools tested, best F1 60% on 6,055 apps","R8-era apps break TPL tools: benchmark reveals 60% F1 ceiling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000948,"raw_usage":{"total_tokens":3877,"prompt_tokens":735,"completion_tokens":3142,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":479,"completion_tokens_details":{"reasoning_tokens":3036}},"tokens_in":479,"tokens_out":3142,"duration_ms":20626,"temperature":1.0,"reasoning_tokens":3036,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:22:42.098633+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compile a fresh, stratified sample of TPL-Benchmark projects spanning DX-era and R8 full-mode AGP versions and compare TPL-Extractor's triplets against Gradle's resolved release-runtime classpath: if exact-match falls materially below the reported 50%, or version mismatches concentrate in modern builds, the version-level trustworthiness claim that anchors every tool score is undermined. A second check: run a non-code-feature detector (resources plus build metadata) on the same 946-app evaluation; if it clears roughly 85% library-level F1, the paper's attribution of failure to code-level method","supporting_citations":[{"cited_title":"Reliable Third-Party Library Detection in Android and its Se- curity Applications","cited_arxiv_id":null,"evidence_quote":"LibScout defines the foundational similarity-based signature approach that the paper evaluates and finds fragile under R8-era shrinking."},{"cited_title":"Scalably De- tecting Third-Party Android Libraries With Two-Stage Bloom Filtering.IEEE Transactions on Software Engi- neering, 49(4):2272–2284, 2023","cited_arxiv_id":null,"evidence_quote":"LibLOOM's two-stage bloom-filter detector posts the best efficiency in the evaluation, setting the speed/accuracy balance reference."},{"cited_title":"Detecting third-party libraries in Android applications with high precision and recall","cited_arxiv_id":null,"evidence_quote":"LibPecker achieves the best library-level F1 (60.15%), setting the ceiling the paper's central claim hinges on."},{"cited_title":"Libscan: Towards more precise third-party library identification for android applications","cited_arxiv_id":null,"evidence_quote":"LibScan's candidate-set version-level detection supplies the comparison point for version discrimination accuracy."},{"cited_title":"Beresford, and Stephan A","cited_arxiv_id":null,"evidence_quote":"LibID's two variants set the version-level F1 reference (~41–49%) and demonstrate the prohibitive runtime of precise matching."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LibHunter, the compiler-aware detector with the highest recall, is the paper's key evidence that optimization-aware methods help."},{"cited_title":"ORLIS: Obfuscation-resilient library detec- tion for Android","cited_arxiv_id":null,"evidence_quote":"ORLIS, an obfuscation-resilient detector, posts the worst F1 (2.26%), anchoring the lower bound of the performance spread."},{"cited_title":"LibRadar: Fast and accurate detection of third-party libraries in Android apps","cited_arxiv_id":null,"evidence_quote":"LibRadar represents the clustering-based category, showing fast but coarse package-structure output is too imprecise for reliable use."},{"cited_title":"LibD: Scal- able and Precise Third-Party Library Detection in An- droid Markets","cited_arxiv_id":null,"evidence_quote":"LibD is the other clustering baseline, evaluated for scalability but limited to ambiguous obfuscated package structures."},{"cited_title":"Bissyandé, Jacques Klein, and Yves Le Traon","cited_arxiv_id":null,"evidence_quote":"AndroZoo supplies the benign and malicious app corpus used for the malware downstream study after TPL stripping."}],"review_version":1}