A pre-registered audit of 102 cases plus 18 papers found artifact-embedded oracles unreliable: 20 of 30 signals also triggered on patched builds, so runnable artifacts rarely confirm the CVE.
Repeatability in computer systems research.Com- munications of the ACM, 59(3):62–69, 2016
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
citation-role summary
extension 1
citation-polarity summary
fields
cs.CR 1years
2026 1verdicts
CONDITIONAL 1roles
extension 1polarities
extend 1representative citing papers
citing papers explorer
-
From Runnable to Verifiable: An Independent Reproducibility Study of LLM/Agent-Driven Vulnerability Validation Artifacts
A pre-registered audit of 102 cases plus 18 papers found artifact-embedded oracles unreliable: 20 of 30 signals also triggered on patched builds, so runnable artifacts rarely confirm the CVE.