REVIEW 4 major objections 5 minor 4 references
CveBinarySheet: A Comprehensive Pre-built Binaries Database for IoT Vulnerability Analysis
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper presents CveBinarySheet, a database of 1,033 prebuilt vulnerable binaries tied to CVEs from 1999 to 2024 and designed for IoT, UEFI, and MCU firmware analysis.
desk verdict A dataset paper with no dataset, and the build pipeline described likely yields patched binaries rather than vulnerable ones — interesting scope, but the core claim is unverifiable and probably wrong. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying object is the database itself, CveBinarySheet: a hierarchical file organization of precompiled binaries plus per-CVE JSON metadata. The layout groups files by component name, version number, CPU architecture, and optimization level, so a researcher can request a precise slice such as vulnerable curl for ARMv7 compiled at O0. The metadata supplies the map from each binary to its CVE, affected function names, version range, and the before/after file changes of the patch, while the accompanying compilation scripts give the recipe to rebuild any sample. This paired structure, binary plus metadata plus build recipe, is what allows the dataset to serve training, benchmarking, similarity analysis, and formal-analysis applications.
What would settle it
Disassemble a random sample of 50 published binaries, locate the function named in the metadata, and check whether the vulnerable code path is present; if a large fraction of samples are already patched or the provided build scripts fail to reproduce the claimed binary, the central promise of the database fails.
Extended reading notes
Core claim
On its own terms, the central discovery is that a large, structured collection of vulnerable binaries can be assembled by taking an existing CVE-oriented dataset and compiling the corresponding third-party source versions into executable test cases. The paper claims 1,033 CVEs are covered, with each binary accompanied by structured metadata: CVE identifier, patch URL, affected function names, affected version range, the binary's exact version, and the file paths and line numbers changed by the patch. Binaries are organized hierarchically by component, version, CPU architecture, and optimization level, and compilation scripts are included so that any sample can be rebuilt. The author's thesis is that this pairing of binary, metadata, and build recipe turns scattered CVE records into concrete objects that BSCA tools can actually be run on.
Load-bearing premise
The entire value of the database rests on the assumption that the source version fetched for each CVE can be compiled on all five architectures into a binary that still contains the exact vulnerable code the CVE describes, and the paper provides no build logs or verification of that fidelity.
Editorial extensions
If this is right
- A binary-analysis tool can be evaluated on the same vulnerable function compiled across five architectures, exposing architecture-specific detection failures.
- Binary-similarity models can be trained with positive pairs of binaries that share a vulnerable function and negative pairs drawn from different components, yielding inexpensive ground-truth labels.
- The before/after patch metadata can be turned into a repair corpus in which each vulnerable binary is paired with the exact source changes that fix it.
- Formal-analysis and fuzzing tools can be benchmarked on real-world third-party components rather than synthetic snippets.
- Researchers can study how compiler optimization level (O0 versus O3) changes the detectability of a vulnerability.
Reading between the lines
- Inference: if the database is maintained with verified build logs and cryptographic hashes, it could become a standard cross-architecture benchmark for vulnerability-localization papers, similar to the role source-level datasets play today.
- Inference: a natural falsification check the paper does not run is to disassemble a sample of its binaries, find the function named in the metadata, and confirm the vulnerable program path is still present.
- Inference: because the claimed coverage is limited to 16 user-space third-party components, the practical scope is narrower than 'IoT firmware' as a whole; kernel, bootloader, and MCU-specific code would require a separate extension.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. CveBinarySheet is presented as a pre-built binaries database for IoT vulnerability analysis, containing 1033 CVE entries (1999–2024) across 16 third-party components and five CPU architectures (x86-64, i386, MIPS, ARMv7, RISC-V64), with each binary compiled at optimization levels O0 and O3. The paper describes the construction pipeline based on Arch Linux's AUR, the metadata schema (CVE ID, patch URL, reaching path, function names, affected versions, binary version, file changes), and a hierarchical classification scheme. It also outlines applications in binary similarity, LLM-based repair, and formal analysis, and closes with limitations and future work.
Significance. If the database existed as described and its binaries faithfully corresponded to the specified vulnerable source revisions, it would be a useful resource for benchmark-driven binary analysis research. The multi-architecture, multi-optimization-level design and the explicit metadata schema are sensible desiderata, and the paper correctly identifies a gap in publicly available vulnerable-binary datasets. However, the significance cannot currently be assessed because the manuscript provides no path to the artifact, no statistics beyond aggregate counts, and no evidence that the compiled binaries contain the claimed vulnerabilities. The paper is therefore, at present, a description of a plan rather than a verifiable dataset contribution.
major comments (4)
- [Sec. II.D and Sec. III.A] The central claim that the database contains binaries faithfully reproducing 1033 CVEs is unsupported and likely false under the described pipeline. Arch Linux's AUR uses a rolling-release model that continuously applies upstream security patches; the paper even cites this as a benefit, but it acts against reproducing historical vulnerable source states. The manuscript does not describe any mechanism for pinning source revisions to the exact vulnerable version matching each CVE, nor does it explain how the AUR packages for CVEs from 1999–2024 would correspond to archived vulnerable snapshots. Consequently, the binaries are expected to be built from current, patched sources rather than the vulnerable revisions, which undermines the dataset's core value.
- [Sections II and IV (entire paper)] No download link, repository URL, archive identifier, checksum, or any other access mechanism for CveBinarySheet is provided anywhere in the manuscript. Without access to the database, reviewers and researchers cannot verify the existence or contents of the claimed 1033 CVE entries, the 16 components, the five architectures, or the two optimization levels. The absence of per-component, per-architecture, or per-version statistics also prevents any sanity check of the aggregate counts. This is a load-bearing omission for a dataset paper.
- [Sec. V (Limitations) and Sec. II.D] There is no verification that the produced executables actually contain the targeted vulnerability. The paper reports no build logs, no compiler/toolchain versions beyond 'O0/O3', no cross-compilation configurations for MIPS, ARMv7, or RISC-V64, and no functional tests showing that the binaries execute or that vulnerable functions are present and reachable. The 'Compilation Consistency' limitation in Sec. V addresses only compiler-version variation, not source-version fidelity, which is the decisive assumption. Without such evidence, the database's fundamental artifact is unverified.
- [Sec. II.A and Sec. III.A] The metadata claims are also unvalidated. Function names and file changes are stated to be extracted via Tree-sitter, but no accuracy, completeness, or manual-audit results are reported, and the relationship between the extracted functions and the listed CVEs is not demonstrated. Additionally, the 'Binary Version' field must be shown to fall within the 'Affected Versions' range for each CVE; the manuscript provides no examples or consistency checks, leaving uncertainty about whether the metadata and binaries actually correspond.
minor comments (5)
- [Abstract and Sec. I] The phrase 'meticulously curated database' is not supported by any reproducibility measures, such as checksums, a data-availability statement, or a versioned release; consider tempering the wording or adding such measures.
- [Reference [1]] The Megavul reference is listed as 'arXiv preprint arXiv:2406.12415, 2023' but the arXiv identifier and year appear inconsistent; the citation should be checked.
- [Sec. VI] Parts of Sec. VI repeat Sec. IV nearly verbatim (e.g., the jTrans description). This duplication should be removed or condensed for clarity.
- [Sec. II.D] The phrase 'Reduced Upstream Patch Interference' is confusing given that a rolling-release model actively introduces upstream patches; the intended meaning should be stated more precisely.
- [Sec. III.C] The description of 'Integration with Existing Categories' is vague and does not explain how a user would actually query or cross-reference binaries, metadata, and scripts; a concrete example or schema snippet would help.
Circularity Check
No circularity: CveBinarySheet is a dataset-resource paper with no derivation, fitted parameters, or self-citation chain that reduces a claimed result to its own inputs.
full rationale
CveBinarySheet makes no predictive or derived claim; the central claim is that the dataset exists and contains the described binaries and metadata. The construction pipeline in Sec. II.D is a procedure description, not an argument whose conclusion is already contained in its premises. No equations or fitted parameters are present, and no quantity is fit to one subset of data and then reported as a prediction of a closely related quantity. The cited prior work (MegaVul) is an external data source rather than the present author's own unverified result, and it is not invoked to forbid alternatives or to justify a uniqueness claim. The limitations section explicitly acknowledges that compiler-version variation may affect binary behavior, which is an honesty about empirical risk rather than a circular step. The AUR rolling-release choice may raise a legitimate correctness concern about whether compiled binaries faithfully reproduce historical vulnerable revisions, but that is a validation and fidelity issue, not circular reasoning. Every load-bearing statement describes the artifact itself instead of deriving a conclusion from an assumption that already contains the conclusion, so no circularity is present.
Assumptions & free parameters
assumptions (4)
- domain assumption AUR packages can supply the exact vulnerable source versions for each CVE.
- domain assumption Tree-sitter extraction of affected function names and file changes is accurate.
- domain assumption Compiling at O0 and O3 preserves the vulnerability in the binary for all five architectures.
- domain assumption The 1033 CVEs and 16 components are representative of the IoT, UEFI, and MCU vulnerability surface.
Cite this review
Pith. "Pith review of CveBinarySheet: A Comprehensive Pre-built Binaries Database for IoT Vulnerability Analysis." pith.science (2026). https://pith.science/paper/Y7KS2AUW
@misc{pith2026250108840,
author = {Pith},
title = {Pith review of: CveBinarySheet: A Comprehensive Pre-built Binaries Database for IoT Vulnerability Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/Y7KS2AUW}},
note = {Machine review of arXiv:2501.08840}
}
read the original abstract
Binary Static Code Analysis (BSCA) is a pivotal area in software vulnerability research, focusing on the precise localization of vulnerabilities within binary executables. Despite advancements in BSCA techniques, there is a notable scarcity of comprehensive and readily usable vulnerability datasets tailored for diverse environments such as IoT, UEFI, and MCU firmware. To address this gap, we present CveBinarySheet, a meticulously curated database containing 1033 CVE entries spanning from 1999 to 2024. Our dataset encompasses 16 essential third-party components, including busybox and curl, and supports five CPU architectures: x86-64, i386, MIPS, ARMv7, and RISC-V64. Each precompiled binary is available at two compiler optimization levels (O0 and O3), facilitating comprehensive vulnerability analysis under different compilation scenarios. By providing detailed metadata and diverse binary samples, CveBinarySheet aims to accelerate the development of state-of-the-art BSCA tools, binary similarity analysis, and vulnerability matching applications.
Reference graph
Works this paper leans on
-
[1]
Megavul: A c/c++ vulnerability dataset with comprehensive code repre sentation,
C. Ni, L. Shen, X. Y ang, Y . Zhu, and S. Wang, “Megavul: A c/c++ vulnerability dataset with comprehensive code repre sentation,” arXiv preprint arXiv:2406.12415 , 2023. [Online]. Available: https://doi.org/10.48550/arXiv.2406.12415
-
[2]
jtrans: Jump-aware transformer for binary co de similarity,
H. Wang, W. Qu, G. Katz, W. Zhu, Z. Gao, H. Qiu, J. Zhuge, and C. Zhang, “jtrans: Jump-aware transformer for binary co de similarity,” 2022, arXiv preprint arXiv:2205.12713. [Onl ine]. Available: https://doi.org/10.48550/arXiv.2205.12713
-
[3]
Are Large Language Models Memorizing Bug Benchmarks?
D. Ramos, C. Mamede, K. Jain, P . Canelas, C. Gamboa, and C. Le Goues, “Are large language models memorizing bug benchmarks?” arXiv preprint arXiv:2411.13323 , 2024, arXiv preprint arXiv:2411.13323. [Online]. Available: https://doi.org/10.48550/arXiv.2411.13323
work page Pith review arXiv doi:10.48550/arxiv.2411.13323 2024
-
[4]
T. Barabosch, “Cwe checker,” https://github.com/fkie -cad/cwe checker, 2023
work page 2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.