{"id":"e946cdc0-ae84-4eaf-a5c3-21a82f9fd49e","arxiv_id":"2603.25997","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Java exploits transferred across library versions identify 83% of known-vulnerable versions (99.3% precision), and manual migration pushes recall to 96.1%.","lead":"Researchers ran 259 published Java security exploits against 28,150 historical library versions to see whether an exploit written for one version can identify other vulnerable versions. They found the exploits flagged 83% of known-vulnerable versions with only 89 false alarms, and manual adaptation raised recall to 96.1%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Manual ground-truth accuracy is the linchpin: high inter-rater agreement does not validate labels, and recall/precision estimates shift if the inducing-commit tracing is wrong.","rationale":"The reader's weakest-assumption analysis identifies the manually constructed ground truth as the key risk, and I agree: every headline number in RQ1 is a comparison to that ground truth. The paper's own evidence for the ground truth is inter-rater agreement (a consistency measure) plus the CPE update (which validates only a small, exploit-positive subset). Neither establishes that the 14,378 affected-version labels are accurate. The proposed test directly audits a representative slice of the ground truth with an independent, behavior-based oracle. If the audit passes, the central claim is substantially strengthened; if it fails, the recall/precision figures and the comparison to databases and SOTA tools all need re-computation. This concern does not require changing the reader's CONDITIONAL verdict—it reinforces the conditionality—so the verdict should remain UNCHANGED. I considered whether source composition (constructed vs. disclosed exploits) is a stronger concern, but the ground-truth audit is more fundamental: even if the exploit set is perfectly representative, unreliable labels would invalidate the measurement. Minor numeric inconsistencies in Section 4.2/5.3 are noted but do not affect this assessment.","tokens_in":20353,"tokens_out":12557,"duration_ms":138828,"concrete_test":"Independently re-annotate a stratified random sample of 20 CVEs (covering roughly 2,000 versions across at least 10 libraries) using an alternative oracle: for each version, build the library from its source tag and run the official regression test from the fixing commit, requiring that the test fails on vulnerable code and passes on fixed code. Compare these labels to the published ground truth. If per-version agreement is below 0.90, or if recomputing version-level recall/precision on the corrected labels shifts the headline numbers by more than 5 percentage points, the ground-truth concern lands and the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—83.0% recall and 99.3% precision for exploit-based affected-version identification—is measured entirely against the manually constructed ground truth of 14,378 affected versions (Section 2.2). No independent accuracy audit of this ground truth is reported. Cohen's Kappa of 0.911 demonstrates that the two annotators apply the same labeling criteria consistently, but it does not demonstrate that those criteria are correct. The procedure in Stage 2 requires tracing vulnerability-inducing commits through refactorings and file renames, and Stage 3 requires judging whether each release artifact 'explicitly preserves the vulnerable logic'—both are error-prone judgments. A single misidentified inducing commit can shift labels for every earlier version of that library, directly changing the numerator and denominator of recall. The 796 CPE confirmations cited as external evidence validate only exploit-identified omissions relative to NVD, not the full manual annotation. Moreover, many exploits in the dataset were constructed from the same patch tests used to identify fixing commits, so exploit success and ground-truth labeling are not fully independent. If the manual labels are systematically wrong—for example, if older versions are labeled vulnerable because the code is present but not actually exploitable—then the reported recall is inflated. This is the load-bearing assumption under the headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the first large-scale empirical study of whether disclosed Java library vulnerability exploits can be applied across library versions. The authors construct a dataset of 259 exploits covering 128 libraries and 28,150 versions, manually annotate 14,378 affected versions, and execute each exploit against the version history. They report that unmodified exploits achieve 83.0% recall and 99.3% precision in identifying affected versions (Finding 1), outperforming most vulnerability databases and static analysis tools. They then analyze failures, propose ten manual migration strategies, and claim that migration raises recall to 96.1% (Finding 4). The paper also contributes 796 confirmed CPE-version omissions. The central claim is that the widely held 'exploits are version-specific' limitation is not valid at scale.","tokens_in":20636,"tokens_out":6723,"duration_ms":66115,"significance":"If the central measurement is trustworthy, the paper makes a substantial contribution: it overturns a frequent assumption about exploit version-specificity, provides the largest reproducible Java-library exploit dataset to date, and offers an externally verified contribution to CPE (796 versions). The strong points are the machine-executed cross-version reproduction protocol, the comparison against five databases, and the concrete external confirmation from the CPE Team. The main weaknesses concern the validity of the manually constructed ground truth, the reuse rather than execution of Vision/V-SZZ in the tool comparison, and several inconsistencies in the reported counts. Overall, the paper is practically relevant and the empirical design is mostly sound, but the headline numbers rest on a ground-truth construction that has not been independently audited.","major_comments":[{"comment":"The manual ground truth is the sole yardstick for all RQ1 metrics. Cohen's Kappa of 0.911 measures inter-rater reliability, not correctness. Stage 2 (inducing-commit tracing through refactorings and renames) and Stage 3 (judging whether each release artifact 'explicitly preserves the vulnerable logic') are both error-prone; a single wrong inducing commit can shift labels for all earlier versions of a library. Moreover, the constructed exploits in Section 2.1.2 are derived from patch tests, and the ground truth is derived from the same fix commits, so the exploit outcomes and the labels are not fully independent. The 796 CPE validations cover only 13 CVEs and confirm omissions relative to NVD, not the full 14,378-version annotation. Please provide an independent accuracy audit, e.g., an external expert review of a random sample, a second independent labeling method on a subset, or a sensi","section":"Section 2.2 and Section 4.2"},{"comment":"The affected-version failure counts are internally inconsistent. Section 2.3 and Figure 2 state that the 2,447 affected versions on which exploits fail consist of 1,769 build failures and 678 reproduction failures. Section 5.2 and Sections 5.3-5.4 instead state 1,775 build-failure cases and 672 exploit-failure cases. Both pairs sum to 2,447, but the individual categories feed the root-cause percentages (e.g., 89.8% library compatibility, 27.5% exploit failure) and the migration-success rates in RQ3. The discrepancy must be reconciled, and the actual counts used for each percentage should be made explicit.","section":"Section 5.2 vs. Section 2.3 / Figure 2"},{"comment":"The claimed comparison with state-of-the-art tools does not actually run Vision or V-SZZ. The paper reuses results from Vision's replication package on a 95-CVE overlap subset, while acknowledging the package is hard-coded for C/C++ projects. The overlap subset may be non-representative: it is filtered by the availability of Vision's published outputs and by dataset intersection. This weakens Finding 2 and the broader claim that exploits 'outperform most... assessment tools.' Please either execute Vision/V-SZZ on the same 28,150-version universe as the exploits, or explicitly restrict the conclusion to a preliminary comparison and provide evidence (CWE distribution, library categories, version counts) that the 95-CVE subset is representative.","section":"Section 4.1.2 and Table 2"},{"comment":"The headline precision statistic is inconsistently reported. Table 1 reports 99.3% precision with 89 false positives; the text says 'Among the 12,018 versions identified as vulnerable, exploits incurred only 89 false positives, achieving a precision of 99.4%.' Given 11,931 true positives + 89 false positives = 12,020, the correct denominator yields 99.3%, not 99.4%, and the number 12,018 is unexplained. Since precision and recall are the paper's central quantitative claims, these numbers must be reconciled.","section":"Section 4.2, precision paragraph"}],"minor_comments":[{"comment":"The arXiv metadata title ('Assessing the Cross-Version Applicability of Java Library Vulnerability Exploits') differs from the title on the first page ('A Large-scale Empirical Study on the Generalizability of Disclosed Java Library Vulnerability Exploits'). Use one consistent title.","section":"Title"},{"comment":"Reference [13] refers to 'CVE-2020-2461' while the text in Section 4.2 discusses 'CVE-2020-24616'; fix the inconsistency.","section":"References"},{"comment":"The legend 'NotIntroduced Fixed/DisclosedVulnerable' is difficult to parse. Clarify the distinction between 'not introduced' and 'fixed/disclosed' and define the columns in the caption or text.","section":"Figure 2"},{"comment":"The cross-review step does not report agreement counts or a Kappa for the random 20-sample validation. Please report how many labels were confirmed and how many required discussion.","section":"Section 5.1.2"},{"comment":"The rows 'Exploit Unique' and 'Source Unique' are not defined in the table caption or in the surrounding text. Define what 'unique' means relative to the compared data source.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is important and the CPE external validation is a strong point. I am not rejecting based on disagreement with the prior 'version-specific' belief. The main issue is evidential: the ground-truth construction is not independently audited and several count inconsistencies affect the headline statistics. If the authors can provide an external accuracy audit on a sample, reconcile the counts, and either run the SOTA tools or clearly scope the comparison, the paper would be publishable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is the largest empirical study to date on cross-version exploit applicability, and the headline result — 83% recall, 99.3% precision without migration — holds up as a measurement. The dataset (259 exploits, 128 libraries, 28,150 versions, 14,378 manually labeled affected versions) is the real contribution. The comparison against five databases and two SOTA tools is careful, and the 796 missing CPE versions confirmed by NVD's team are strong external evidence that exploit execution finds real omissions.\n\nThe paper does what it claims: prior work (Dai, Wu) had much smaller samples and didn't systematically quantify applicability. The 10-strategy migration taxonomy and 1,885-case benchmark are useful, though the migration itself is manual and the cost numbers (time/LoC) are rough. The 96.1% recall after migration is an upper bound, not a claim about automation.\n\nSoft spots, in proportion. The biggest is the ground-truth construction. Stage 2 and Stage 3 require tracing inducing commits through refactors and judging whether each release artifact \"explicitly preserves the vulnerable logic.\" Cohen's Kappa of 0.911 shows the two annotators are consistent, but consistency isn't correctness. An independent accuracy audit on a random sample — say, comparing against a third-party tool or a second manual pass on a subset — would harden the claim. Without it, the 83% and 99.3% are conditional on the labels being right. That said, I don't think this is a fatal flaw: the exploit execution results are independent of the annotations in the sense that the 12,020 successfully reproduced versions are an empirical fact, and the CPE confirmations ground part of the output. A systematic error large enough to flip the conclusion would require the annotators to be wrong in a very consistent way.\n\nThere are also minor numeric inconsistencies (1,769 vs. 1,775 build failures; 678 vs. 672 exploit failures) and the preprint doesn't directly link the replication package, though it's referenced. The exploit sample skews toward publicly available and patch-derived payloads; the paper acknowledges this in Section 7.2, but it means the 83% is for that population, not all Java vulnerabilities.\n\nBottom line: send it to peer review. The dataset alone is worth referees' time, and the central finding is important even if the headline numbers need to be re-verified. I'd ask the authors for the artifact and an independent audit of the ground truth, and I'd expect heavy but manageable revision. I'd cite it if I work on this area.","headline":"Large empirical study that mostly disproves the 'exploits are version-specific' assumption; the dataset alone justifies a serious look, despite a manually built ground truth that needs an independent audit.","tokens_in":21103,"tokens_out":2831,"would_cite":true,"duration_ms":28862,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A disclosed exploit, run unchanged against a library's version history, triggers the vulnerability on 83.0% of affected versions at 99.3% precision, contradicting the claim that exploits are version-specific.","keywords":["supply chain security","Java libraries","vulnerability exploits","affected version identification","cross-version applicability","exploit migration","empirical software engineering","vulnerability databases"],"falsifier":"Re-run the experiment on an independently constructed ground truth: take a random sample of 200 of the 14,378 affected-version labels, download the exact release artifact, and run the disclosed exploit under the paper's stated environment. If the exploit fails on many labels that were supposed to be confirmed vulnerable, or if a fresh audit of the fixing commit shows the fix was present earlier than annotated, the reported recall is inflated. The most decisive single check is whether the 796 versions the authors report as newly confirmed actually exhibit the vulnerable code path when opened.","tokens_in":20265,"feed_emoji":"🛡️","tokens_out":7195,"duration_ms":74386,"temperature":0.7,"pith_summary":"The paper challenges the widely repeated claim that vulnerability exploits only work on the exact version they were written for. By executing 259 disclosed exploits against 28,150 historical versions of 128 Java libraries and comparing results to a manually verified affected-version list, the authors find that unmodified exploits identify 83.0% of affected versions (11,931 of 14,378) with 99.3% precision, outperforming most major vulnerability databases. When they manually adapt the exploits that failed, recall rises to 96.1%, and they condense those adaptations into ten reusable migration strategies. A sympathetic reader cares because exploit execution could serve as a concrete, low-false-positive complement to metadata-based vulnerable-version databases and static analysis tools.","feed_headline":"Disclosed exploits reach 83% of vulnerable Java versions","feed_subtitle":"A 28,150-version study shows exploit execution beats most databases at 99.3% precision","key_machinery":"The load-bearing mechanism is a three-stage execution-and-alignment pipeline. It starts with a set of 259 reproduced exploits, then builds a manually verified list of 14,378 affected versions by tracing the commit that patched each vulnerability back to the commit that introduced it and inspecting actual release artifacts, and finally executes every exploit across all historical versions with only the library version changed. The behavior outcomes are aligned against the manual list and categorized as aligned, build failure, exploit failure, or unexpected behavior. The migration step compares failing versions with nearby working versions using source diffs, yielding a taxonomy of ten adaptat","core_discovery":"The central claim, on the paper's own terms, is that disclosed Java library vulnerability exploits exhibit high cross-version applicability: executing an exploit with only the library version switched reproduces the vulnerable behavior on 83.0% of truly affected versions, with a false-positive rate low enough to put it ahead of most vulnerability databases and comparable in recall to the strongest static-analysis approach while being far more precise. The paper further claims that most of the remaining failures are mechanical—breaking changes during library evolution, such as renamed methods and missing classes, and environmental mismatches such as JDK versions—and that adapting exploits usi","pith_inferences":["Inference: If these Java results generalize, exploit-driven version assessment could be exported to other package ecosystems with similar release semantics; the paper's data is Java-only, so this remains a hypothesis.","Inference: The finding that most false negatives are mechanical suggests a practical product architecture: a continuously running regression that smoke-tests each new library release against known exploits, catching reintroduced vulnerable behavior before advisory lists update.","Inference: The 562 unmigratable cases—fuzzing-generated payloads and semantic logic changes—mark the research frontier; solving those is less about API renames and more about understanding intended behavior, which may require semantic program analysis or large-language-model reasoning.","Inference: From a security-tool user's standpoint, a failed exploit should be treated as \"no evidence of exploitability,\" not \"safe,\" because the false-negative rate is roughly 17% even before considering environment differences."],"forward_implications":["Vulnerability databases can treat disclosed exploits as a validation tool: in this study, exploit execution identified 796 affected versions that were missing from a widely used affected-version dictionary and confirmed by the dictionary's maintainers.","Developers and scanners should not rely on a single metadata source; even the best database evaluated still missed over a thousand versions that exploits found, and exploits in turn miss versions that databases list.","Because exploit execution achieves 99.3% precision, it can cut the manual verification burden for reported affected versions.","Diff-guided migration resolves most failures (77.1%), suggesting that automated exploit migration is achievable by combining code-change analysis with environment adjustment.","Exploit-based assessment has a ceiling: after migration, 562 affected versions remain unreachable, so it supplements—not replaces—existing databases and tools."],"fun_headline_variants":["Exploit execution hits 83% recall, 99.3% precision on Java CVEs","Disclosed exploits verify 28K Java versions with high precision","Study: exploit execution beats vulnerability databases on Java","Migration boosts exploit recall on Java CVEs to 96.1%","Java exploits confirm 796 missing CPE affected versions"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The one premise the whole study leans on is that the manually built ground-truth list of 14,378 affected versions is correct: if even a few percent of those labels are wrong—versions marked vulnerable that are actually patched, or patched versions marked vulnerable—the 83% recall and 99.3% precision numbers move by the same few percent, and the comparisons with databases and tools shift.","fun_headline_variants_meta":{"raw":{"variants":["Exploit execution hits 83% recall, 99.3% precision on Java CVEs","Disclosed exploits verify 28K Java versions with high precision","Study: exploit execution beats vulnerability databases on Java","Migration boosts exploit recall on Java CVEs to 96.1%","Java exploits confirm 796 missing CPE affected versions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1176,"prompt_tokens":807,"completion_tokens":369,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":279}},"tokens_in":551,"tokens_out":369,"duration_ms":3645,"temperature":1.0,"reasoning_tokens":279,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T17:18:22.093720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiment on an independently constructed ground truth: take a random sample of 200 of the 14,378 affected-version labels, download the exact release artifact, and run the disclosed exploit under the paper's stated environment. If the exploit fails on many labels that were supposed to be confirmed vulnerable, or if a fresh audit of the fixing commit shows the fix was present earlier than annotated, the reported recall is inflated. The most decisive single check is whether the 796 versions the authors report as newly confirmed actually exhibit the vulnerable code path when opened.","supporting_citations":[],"review_version":1}