{"id":"00367341-0ca5-4339-8c16-eea0bef675d5","arxiv_id":"2501.08840","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"A dataset paper that proposes prebuilt vulnerable binaries for 1,033 CVEs across 16 components and 5 architectures, but supplies no way to access or verify the data.","lead":"This paper describes a database of pre-compiled binaries for 1,033 historical CVEs, but it provides no download link or validation. Security researchers studying binary vulnerabilities might find such a database useful if it existed in accessible form.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dataset's core value depends on binaries that actually contain the stated vulnerability, but the AUR-based pipeline (Sec. II.D) provides no source-version pinning and likely yields patched current sources, so the claimed artifacts are unverified and probably unfaithful.","rationale":"The reader's weakest assumption is that source versions fetched via Arch Linux's AUR can be compiled into binaries that still contain the targeted vulnerability. My stress-test agrees and sharpens the concern: the paper's own justification for AUR (Section II.D) emphasizes rolling releases and reduced upstream patch interference, but patched current sources are the opposite of what a vulnerable-binary dataset requires. No mechanism is described to check out the pre-patch revision corresponding to each CVE. This is not merely a reproducibility gap; it undermines the defining property of the proposed resource. A binary compiled from current AUR sources may be useful for other purposes, but it is not a faithful vulnerable binary for a historical CVE. The paper also provides no download link or artifact repository, so even the existence of the claimed 1033-entry dataset cannot be independently checked. The detailed taxonomy in Section III is a reasonable organizational plan, but a dataset paper's central contribution is the dataset itself, and that contribution is neither accessible nor validated. I therefore keep the reader's REJECT verdict without modification.","tokens_in":4816,"tokens_out":3531,"duration_ms":41610,"concrete_test":"Request from the authors one complete sample artifact: the PKGBUILD or source manifest and compiled binaries for a single CVE with a public PoC (e.g., a busybox or curl CVE). Check that the source version or commit matches the NVD affected range, rebuild with the stated flags, and run the PoC against the O0 and O3 binaries. If the PoC does not trigger on at least the O0 binary, the compilation pipeline does not preserve the vulnerability; the absence of any such artifact would also falsify the current availability claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CveBinarySheet supplies pre-built binaries that faithfully correspond to 1033 CVEs. For this to hold, each binary must be compiled from the exact vulnerable source revision for its CVE, with no security patch applied during the build. The paper never describes how vulnerable source revisions are selected or pinned. Section II.D instead justifies Arch Linux's AUR by its rolling-release model, which continuously applies upstream security patches; that property works against reproducing historical vulnerable states, not for them. AUR provides current PKGBUILDs and patched source trees, not archived snapshots for CVEs spanning 1999-2024. The paper also omits toolchain versions, cross-compilation setup for MIPS, ARMv7, and RISC-V64, build logs, and any verification that the resulting executables actually trigger the corresponding CVE. Section V's 'Compilation Consistency' limitation addresses compiler-version variation, not source-version fidelity, which is the decisive assumption. Consequently, the database's central artifact is unverified and, under the described pipeline, likely unfaithful.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"CveBinarySheet is presented as a pre-built binaries database for IoT vulnerability analysis, containing 1033 CVE entries (1999–2024) across 16 third-party components and five CPU architectures (x86-64, i386, MIPS, ARMv7, RISC-V64), with each binary compiled at optimization levels O0 and O3. The paper describes the construction pipeline based on Arch Linux's AUR, the metadata schema (CVE ID, patch URL, reaching path, function names, affected versions, binary version, file changes), and a hierarchical classification scheme. It also outlines applications in binary similarity, LLM-based repair, and formal analysis, and closes with limitations and future work.","tokens_in":5006,"tokens_out":2197,"duration_ms":22982,"significance":"If the database existed as described and its binaries faithfully corresponded to the specified vulnerable source revisions, it would be a useful resource for benchmark-driven binary analysis research. The multi-architecture, multi-optimization-level design and the explicit metadata schema are sensible desiderata, and the paper correctly identifies a gap in publicly available vulnerable-binary datasets. However, the significance cannot currently be assessed because the manuscript provides no path to the artifact, no statistics beyond aggregate counts, and no evidence that the compiled binaries contain the claimed vulnerabilities. The paper is therefore, at present, a description of a plan rather than a verifiable dataset contribution.","major_comments":[{"comment":"The central claim that the database contains binaries faithfully reproducing 1033 CVEs is unsupported and likely false under the described pipeline. Arch Linux's AUR uses a rolling-release model that continuously applies upstream security patches; the paper even cites this as a benefit, but it acts against reproducing historical vulnerable source states. The manuscript does not describe any mechanism for pinning source revisions to the exact vulnerable version matching each CVE, nor does it explain how the AUR packages for CVEs from 1999–2024 would correspond to archived vulnerable snapshots. Consequently, the binaries are expected to be built from current, patched sources rather than the vulnerable revisions, which undermines the dataset's core value.","section":"Sec. II.D and Sec. III.A"},{"comment":"No download link, repository URL, archive identifier, checksum, or any other access mechanism for CveBinarySheet is provided anywhere in the manuscript. Without access to the database, reviewers and researchers cannot verify the existence or contents of the claimed 1033 CVE entries, the 16 components, the five architectures, or the two optimization levels. The absence of per-component, per-architecture, or per-version statistics also prevents any sanity check of the aggregate counts. This is a load-bearing omission for a dataset paper.","section":"Sections II and IV (entire paper)"},{"comment":"There is no verification that the produced executables actually contain the targeted vulnerability. The paper reports no build logs, no compiler/toolchain versions beyond 'O0/O3', no cross-compilation configurations for MIPS, ARMv7, or RISC-V64, and no functional tests showing that the binaries execute or that vulnerable functions are present and reachable. The 'Compilation Consistency' limitation in Sec. V addresses only compiler-version variation, not source-version fidelity, which is the decisive assumption. Without such evidence, the database's fundamental artifact is unverified.","section":"Sec. V (Limitations) and Sec. II.D"},{"comment":"The metadata claims are also unvalidated. Function names and file changes are stated to be extracted via Tree-sitter, but no accuracy, completeness, or manual-audit results are reported, and the relationship between the extracted functions and the listed CVEs is not demonstrated. Additionally, the 'Binary Version' field must be shown to fall within the 'Affected Versions' range for each CVE; the manuscript provides no examples or consistency checks, leaving uncertainty about whether the metadata and binaries actually correspond.","section":"Sec. II.A and Sec. III.A"}],"minor_comments":[{"comment":"The phrase 'meticulously curated database' is not supported by any reproducibility measures, such as checksums, a data-availability statement, or a versioned release; consider tempering the wording or adding such measures.","section":"Abstract and Sec. I"},{"comment":"The Megavul reference is listed as 'arXiv preprint arXiv:2406.12415, 2023' but the arXiv identifier and year appear inconsistent; the citation should be checked.","section":"Reference [1]"},{"comment":"Parts of Sec. VI repeat Sec. IV nearly verbatim (e.g., the jTrans description). This duplication should be removed or condensed for clarity.","section":"Sec. VI"},{"comment":"The phrase 'Reduced Upstream Patch Interference' is confusing given that a rolling-release model actively introduces upstream patches; the intended meaning should be stated more precisely.","section":"Sec. II.D"},{"comment":"The description of 'Integration with Existing Categories' is vague and does not explain how a user would actually query or cross-reference binaries, metadata, and scripts; a concrete example or schema snippet would help.","section":"Sec. III.C"}],"recommendation":"reject","confidential_remarks":"The manuscript currently functions as a project proposal rather than a completed dataset paper. The central artifact is inaccessible, and the few concrete details that are given (AUR-based rolling-source builds without version pinning) point toward binaries that would not faithfully represent the targeted CVEs. These issues are unlikely to be fixable within a revision unless the authors rebuild and validate the entire dataset, provide access, and publish per-CVE build provenance. If a future version includes the actual database with verification, this could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a proposal for a vulnerable-binary database, not a database. There's no download link, no hashes, no build logs, nothing a reviewer can touch. And the described build method—Arch Linux's AUR—works against the stated goal: rolling release means you get the latest patched sources, not the vulnerable revision the CVE refers to.\n\nI'll credit what's genuinely there. The domain need is real: pre-built vulnerable binaries across architectures are scarce, and a dataset with 1033 CVEs, 16 components, five architectures, and O0/O3 builds would be useful. The metadata schema (CVE ID, patch URL, affected functions, file changes) is sensible. Including RISC-V64 is a thoughtful touch.\n\nBut the soft spots are structural. First, no access. The paper says 'database' but gives no URL. A dataset paper without the dataset is unverifiable prose. Second, no validation that the binaries actually contain the vulnerability. The authors never show a single binary that triggers a CVE. Third, and most damaging, the choice of AUR is defended with claims that contradict the goal. Section II.D says the rolling release model 'reduces upstream patch interference' and preserves reproducibility. In practice, AUR gives you current versions with security patches applied. The paper never explains how vulnerable source revisions are pinned or checked out, so even if the binaries exist, they are likely patched and useless for vulnerability research. The limitation section talks about compiler consistency, not source-version fidelity, which is the assumption that actually carries the whole project.\n\nThere's also no comparison to existing binary datasets, so the novelty claim is weak. The references are thin; the related work mostly repeats the same paragraph structure.\n\nWho gets value from this? Someone might use the proposed metadata schema as a starting point, but nobody can use the dataset because there isn't one. It's a two-page idea stretched to seven.\n\nMy recommendation: if this lands on your desk, send it out for review—the area matters and the reviewers can quickly tell the author what's missing—but expect the verdict to be reject unless a follow-up includes the actual artifacts, verified vulnerable builds, and a completely different method for obtaining vulnerable source snapshots.","headline":"A dataset paper with no dataset, and the build pipeline described likely yields patched binaries rather than vulnerable ones — interesting scope, but the core claim is unverifiable and probably wrong.","tokens_in":5477,"tokens_out":4109,"would_cite":false,"duration_ms":41357,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper presents CveBinarySheet, a database of 1,033 prebuilt vulnerable binaries tied to CVEs from 1999 to 2024 and designed for IoT, UEFI, and MCU firmware analysis.","keywords":["CVE","IoT Security","Binary Analysis","Vulnerability Database","Pre-built Binaries","Firmware Security","BSCA","Binary Similarity"],"falsifier":"Disassemble a random sample of 50 published binaries, locate the function named in the metadata, and check whether the vulnerable code path is present; if a large fraction of samples are already patched or the provided build scripts fail to reproduce the claimed binary, the central promise of the database fails.","tokens_in":4635,"feed_emoji":"🛡️","tokens_out":9493,"duration_ms":88707,"temperature":0.7,"pith_summary":"The pith of this paper is that the missing piece for binary-level vulnerability research is not new analysis algorithms but usable test data: prebuilt binaries that are known to contain a specific CVE. The author presents CveBinarySheet, a database of 1,033 CVE entries from 1999 to 2024, built from 16 widely used third-party components and compiled for five CPU architectures (x86-64, i386, MIPS, ARMv7, RISC-V64), each at two optimization levels (O0 and O3). Every entry carries metadata linking the binary to its CVE, affected functions, affected version range, patch URL, and the file-level changes made by the fix. If the database is faithful, it gives researchers a common substrate for training and benchmarking binary static analysis, similarity matching, and automated repair tools across realistic firmware-relevant environments.","feed_headline":"1,033 pre-built vulnerable binaries now cover five CPU architectures","feed_subtitle":"Researchers get ready-made binaries and metadata for training detection and repair tools across five architectures.","key_machinery":"The carrying object is the database itself, CveBinarySheet: a hierarchical file organization of precompiled binaries plus per-CVE JSON metadata. The layout groups files by component name, version number, CPU architecture, and optimization level, so a researcher can request a precise slice such as vulnerable curl for ARMv7 compiled at O0. The metadata supplies the map from each binary to its CVE, affected function names, version range, and the before/after file changes of the patch, while the accompanying compilation scripts give the recipe to rebuild any sample. This paired structure, binary plus metadata plus build recipe, is what allows the dataset to serve training, benchmarking, similarity analysis, and formal-analysis applications.","core_discovery":"On its own terms, the central discovery is that a large, structured collection of vulnerable binaries can be assembled by taking an existing CVE-oriented dataset and compiling the corresponding third-party source versions into executable test cases. The paper claims 1,033 CVEs are covered, with each binary accompanied by structured metadata: CVE identifier, patch URL, affected function names, affected version range, the binary's exact version, and the file paths and line numbers changed by the patch. Binaries are organized hierarchically by component, version, CPU architecture, and optimization level, and compilation scripts are included so that any sample can be rebuilt. The author's thesis is that this pairing of binary, metadata, and build recipe turns scattered CVE records into concrete objects that BSCA tools can actually be run on.","pith_inferences":["Inference: if the database is maintained with verified build logs and cryptographic hashes, it could become a standard cross-architecture benchmark for vulnerability-localization papers, similar to the role source-level datasets play today.","Inference: a natural falsification check the paper does not run is to disassemble a sample of its binaries, find the function named in the metadata, and confirm the vulnerable program path is still present.","Inference: because the claimed coverage is limited to 16 user-space third-party components, the practical scope is narrower than 'IoT firmware' as a whole; kernel, bootloader, and MCU-specific code would require a separate extension."],"forward_implications":["A binary-analysis tool can be evaluated on the same vulnerable function compiled across five architectures, exposing architecture-specific detection failures.","Binary-similarity models can be trained with positive pairs of binaries that share a vulnerable function and negative pairs drawn from different components, yielding inexpensive ground-truth labels.","The before/after patch metadata can be turned into a repair corpus in which each vulnerable binary is paired with the exact source changes that fix it.","Formal-analysis and fuzzing tools can be benchmarked on real-world third-party components rather than synthetic snippets.","Researchers can study how compiler optimization level (O0 versus O3) changes the detectability of a vulnerability."],"supporting_citations":[{"why":"Supplies the underlying CVE-oriented dataset that CveBinarySheet extends by adding compiled binaries and metadata.","marker":"[1]"},{"why":"Provides the transformer-based binary similarity model named as a target application for the dataset's training and evaluation.","marker":"[2]"},{"why":"Motivates the repair-corpus use case by showing how large language models can memorize bug benchmarks.","marker":"[3]"},{"why":"Gives an example formal-analysis tool that could be benchmarked against the provided binaries.","marker":"[4]"}],"fun_headline_variants":["1,033 CVEs, 5 architectures: prebuilt vulnerable binaries","From 1999 to 2024: 1,033 ready-made vulnerable binaries","Five CPU architectures, 1,033 vulnerable binaries, one database","Vulnerable binaries for 5 architectures: 1,033 CVEs from 1999-2024"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire value of the database rests on the assumption that the source version fetched for each CVE can be compiled on all five architectures into a binary that still contains the exact vulnerable code the CVE describes, and the paper provides no build logs or verification of that fidelity.","fun_headline_variants_meta":{"raw":{"variants":["1,033 CVEs, 5 architectures: prebuilt vulnerable binaries","From 1999 to 2024: 1,033 ready-made vulnerable binaries","Five CPU architectures, 1,033 vulnerable binaries, one database","Vulnerable binaries for 5 architectures: 1,033 CVEs from 1999-2024"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001805,"raw_usage":{"total_tokens":7073,"prompt_tokens":878,"completion_tokens":6195,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":6106}},"tokens_in":494,"tokens_out":6195,"duration_ms":43257,"temperature":1.0,"reasoning_tokens":6106,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:16:02.888613+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Disassemble a random sample of 50 published binaries, locate the function named in the metadata, and check whether the vulnerable code path is present; if a large fraction of samples are already patched or the provided build scripts fail to reproduce the claimed binary, the central promise of the database fails.","supporting_citations":[{"cited_title":"Are Large Language Models Memorizing Bug Benchmarks?","cited_arxiv_id":"2411.13323","evidence_quote":"Motivates the repair-corpus use case by showing how large language models can memorize bug benchmarks."},{"cited_title":"Cwe checker,","cited_arxiv_id":null,"evidence_quote":"Gives an example formal-analysis tool that could be benchmarked against the provided binaries."}],"review_version":1}