{"id":"c4571754-2b9a-4f83-b3ed-01efd8c4cbd9","arxiv_id":"1908.05459","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new open dataset and toolset enables machine-learning classifiers to identify CPU architecture and endianness of ELF binaries with over 98 percent accuracy on code sections and 99.2 percent on complete files.","lead":"This paper builds and releases a large open dataset and toolset for detecting CPU architecture and endianness from raw binary code, and it reproduces state-of-the-art accuracies above 98 percent on code sections. It also reports 99.2 percent accuracy when training on complete binary files, though that result may partly come from reading headers rather than code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 99.2% complete-binary result is not controlled for ELF-header leakage: byte-frequency features include e_machine, which encodes the ground-truth label; no stripped-header or non-ELF test supports the 'arbitrary binaries' claim.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing gap: the complete-binary experiment in Section 4.3 lacks a stripped-header control, so the 99.2% accuracy cannot be cleanly attributed to ISA detection of code. The manuscript itself flags the ELF-only scope in Sections 3.1.1 and 4.3.1, which strengthens rather than weakens this concern. The code-section-only results and the released dataset/toolset are independent contributions with real value, and the paper's reconstruction of prior work is consistent with Clemens and De Nicolao. The correct disposition is to keep the reader's CONDITIONAL verdict: the complete-binary claim needs an additional control or a narrowed claim, but the core methodological contributions do not collapse. No change to the reader's verdict is needed.","tokens_in":21134,"tokens_out":4779,"duration_ms":51180,"concrete_test":"Re-run the Section 4.3 / Table 9 Random Forest (Azure) experiment after zeroing out the ELF header fields e_machine, EI_CLASS, EI_DATA, and all program/section header table bytes in every training and test binary before computing the 293 features, leaving code and data sections otherwise intact. If accuracy remains near 99.2%, the result reflects code/data content; if it drops toward the 90.1% code-only-trained complete-binary accuracy reported in Table 7, the original 99.2% was substantially explained by reading the ELF header.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central novel claim (abstract; contribution (c); Section 4.3, Table 9) is that training and testing on complete binaries identifies ISA with 99.2% accuracy using Random Forest. The load-bearing unstated assumption is that features computed over a complete ELF file reflect object code, not container metadata. Every ELF header carries e_machine, EI_CLASS, and EI_DATA, which directly encode the ground-truth architecture, wordsize, and endianness. The 256 byte-frequency features are computed over the entire file, so those header bytes are part of the feature vector and are highly discriminative. The paper never strips, masks, or ablates header bytes, and it provides no control using headerless object code or non-ELF formats. Section 3.1.1 explicitly limits experiments to ELF, while Section 4.3.1 concedes that file-format structural differences could influence the \"complete binary\" method and that non-ELF evaluation is only future work. Therefore, the 99.2% accuracy cannot be attributed to code analysis rather than header reading, and the title's \"arbitrary binary files and object code sequences\" is unsupported for this claim. The code-section-only results in Section 4.1 remain credible because they use extracted code sections, but they do not validate the complete-binary headline. This is a missing experimental control, not an internal inconsistency.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a toolset, dataset, and web-service API for automated identification of CPU architecture and endianness from binary files and object code. The authors reconstruct and cross-validate two prior methods (Clemens 2015; De Nicolao et al. 2018) using 67,285 ELF code-section samples spanning 23 architectures and multiple classifiers, reporting over 98% accuracy. They also propose training classifiers on complete ELF binaries rather than only code sections, reporting 99.2% accuracy for a Random Forest classifier in Azure. The paper claims this supports usable detection for \"arbitrary binary files and object code sequences\" and releases all artifacts as open source.","tokens_in":21352,"tokens_out":5205,"duration_ms":51287,"significance":"The strongest contribution is reproducibility infrastructure: the paper appears to be the first to publicly release the dataset, extraction toolset, and models for this problem, and the code-section evaluation provides an independent confirmation of prior results on a larger and more balanced corpus. If the code-section claims hold, they usefully validate byte-frequency and signature features across dataset construction choices. The complete-binary result would be a practical convenience, but its current experimental basis does not support the paper's headline generalization; the missing header-ablation control is a fixable experimental gap rather than an internal inconsistency.","major_comments":[{"comment":"The 99.2% complete-binary claim lacks a control for ELF header leakage. The 256 byte-frequency features are computed over the entire file, and the ELF header directly contains e_machine, EI_CLASS, and EI_DATA, which encode the ground-truth architecture, word size, and endianness. No experiment strips, masks, zeroes, or otherwise ablates these header bytes, and no headerless or non-ELF corpus is tested. The reported accuracy therefore cannot be attributed to analysis of object code rather than to reading container metadata. The authors should add an ablation (e.g., zeroed or stripped headers, or code-section-only evaluation under the same training/test setup) and restate the complete-binary conclusion accordingly.","section":"Section 4.3, Table 9"},{"comment":"The title and contribution (c) claim applicability to \"arbitrary binary files and object code sequences,\" but all complete-binary experiments use only Debian-built ELF files. The authors explicitly concede in Section 4.3.1 that file-format structural differences could influence the complete-binary method and that non-ELF evaluation is only future work. Without an out-of-distribution test (e.g., non-Debian ELF, PE/Mach-O, firmware images, or extracted headerless fragments), the generalization claim is not supported. The claims should either be scoped to ELF binaries or substantiated with additional experiments.","section":"Sections 3.1.1 and 4.3.1"}],"minor_comments":[{"comment":"The mips class is listed as upsampled from 418 to 2997 samples, but the resampling procedure is not described; if samples are duplicated, 10-fold cross-validation may be optimistically biased, so the method should be stated explicitly.","section":"Section 3.1, Table 2"},{"comment":"The column headers for the accuracy results are duplicated and misaligned (\"Accuracy BFD+endian\" appears twice along with the Clemens comparison); please relabel the columns to distinguish this paper's BFD, BFD+endian, and prior-work results.","section":"Table 5"},{"comment":"The text says the powerpc/powerpcspe confusion matrices \"can be seen in Table 8,\" but Table 8 reports Azure performance; the matrices are actually in Table 11, so the cross-reference should be corrected.","section":"Section 4.4"},{"comment":"The row \"Average (De Nicolao et al. [15])\" appears to quote that paper's published metrics rather than metrics recomputed on the authors' own 500-binaries-per-architecture test set; the paper should state explicitly whether this is a direct reproduction or a literature comparison.","section":"Section 4.2, Table 7"},{"comment":"References [46] and [47] cite the same SoK paper; this duplicate should be removed or one of the entries should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the code-section experiments and the open dataset/toolset are solid contributions with clear reproducibility value. The 99.2% complete-binary headline, however, should not be presented as evidence for arbitrary-binary detection until a header-ablation or non-ELF control is supplied. I see this as a fixable experimental gap, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper's solid core is the reconstruction of Clemens and De Nicolao's code-section classifiers on a much larger, balanced dataset (67,285 ELF samples, 23 architectures) with consistent >98% accuracy across many classifiers. That part is credible and useful, and the release of the toolset and dataset is genuinely the first public resource of its kind in this subfield. The powerpcspe signature tweak is minor but sensible. The complete-binary training idea is the novel piece, but the headline result is not trustworthy as presented.\n\nThe problem is target leakage. In Section 4.3 (Table 9), the classifiers are trained and tested on full ELF files using byte-frequency features computed over the whole file. The ELF header contains e_machine (and EI_CLASS/EI_DATA) which directly encode the ground-truth architecture, wordsize, and endianness. Nothing in the paper strips, masks, or ablates those header bytes. So the 99.2% accuracy is very plausibly just the classifier reading the header, not learning anything about code. A simple control with header bytes zeroed or removed would settle this; its absence makes the claim unsupported. The paper actually concedes in Section 4.3.1 that file-format structural differences could influence the method and that non-ELF evaluation is only future work, which makes the title's \"arbitrary binary files\" an overstatement.\n\nThe code-section experiments do not have this problem because features come from extracted code sections only and labels come from headers that are not part of the input. That is why the paper is conditionally acceptable rather than rejectable: the central reproduction is sound, one specific novel claim needs a major experimental control, and the dataset remains valuable regardless.\n\nIf I were refereeing, I would ask for a revision that adds a stripped-header control for the complete-binary experiment, tests on at least one non-ELF format or on genuinely headerless fragments, and recalibrates the abstract/title to match what is actually shown. The code-section results can be published as-is. This is a case where serious peer review is warranted, not desk rejection.","headline":"The code-section reproduction and the open dataset are the real contributions; the 99.2% complete-binary headline is not controlled for ELF header leakage.","tokens_in":21980,"tokens_out":1874,"would_cite":false,"duration_ms":18111,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A classifier trained on whole binary files identifies CPU architecture with 99.2% accuracy; code-section-only training still exceeds 98%.","keywords":["binary code analysis","instruction set architecture","endianness","machine learning classification","byte frequency distribution","firmware analysis","random forest","reverse engineering"],"falsifier":"Zero out or remove the architecture field from the headers of the complete-binary test set and repeat the 99.2% experiment; if accuracy falls sharply, the classifier was using header metadata rather than code bytes.","tokens_in":20845,"feed_emoji":"🔍","tokens_out":7413,"duration_ms":67309,"temperature":0.7,"pith_summary":"Wrong guesses about a binary's CPU architecture cause about 10% of failures in IoT firmware analysis, so tools that work on raw object code need a reliable way to name the instruction set. This paper builds an open pipeline, dataset, and web service for that task, and reconstructs two earlier methods to show they hold up. Its own contribution is to train classifiers on complete executable files rather than only code sections; a Random Forest trained this way identifies the architecture with 99.2% accuracy. On code sections alone, all tested classifiers reach over 98% accuracy, matching previous work. The authors also add two signatures that separate two closely related PowerPC variants.","feed_headline":"Full-binary training identifies CPU architecture with 99.2% accuracy","feed_subtitle":"Classifiers trained on whole executables, not just code sections, beat prior ISA detection results.","key_machinery":"The load-bearing object is a 293-dimensional feature vector: 256 byte-frequency values, 4 endianness marker signatures, 31 function prologue and epilogue byte signatures, and 2 new powerpcspe signatures. The byte frequencies give each architecture a characteristic distribution, the endianness markers capture byte-order conventions, and the prologue and epilogue signatures capture coding idioms that differ from CPU to CPU. The paper's novel move is to compute these features over the whole executable file rather than the code section only, and to train the classifiers the same way; the Random Forest that does this reaches the paper's headline 99.2% accuracy.","core_discovery":"The paper's central claim is that byte-level statistics of a complete executable file are enough to identify both the instruction set architecture and the endianness, and that the classifier works even better when trained on whole files instead of extracted code sections. In a 23-architecture corpus of 67,285 samples, a Random Forest trained and tested on complete binaries reached 99.2% accuracy, the best result in the paper. When training and testing were restricted to code sections, every classifier still exceeded 98% accuracy, which the paper presents as an independent confirmation of previous findings. The paper also contributes two byte-signatures for the powerpcspe architecture that raise the F1 score for PowerPC classes by about 2 percentage points.","pith_inferences":["If the 99.2% complete-file accuracy comes largely from reading the architecture field in the file header, the method may not transfer to headerless shellcode; a strip-the-header test would settle this.","The same feature pipeline could be extended to estimate word size and byte order as separate outputs, letting analysts label a binary before a disassembler is chosen.","The released dataset and API make it possible to crowd-source corrections, which could turn accuracy gains from retraining on real-world misclassified samples.","If the complete-binary training trick works for the executable format tested here, it likely carries over to other container formats such as PE or Mach-O, provided code sections are extracted consistently."],"forward_implications":["Architecture identification on complete executable files can reach 99.2% accuracy with a Random Forest, which is high enough for automated triage of firmware and malware corpora.","Code-section-only classifiers match prior results above 98%, so the byte-frequency-plus-signature approach generalizes beyond the original datasets.","Fragments as short as 128 bytes are classified with roughly 90% accuracy by several classifiers, extending the method toward shellcode and memory-dump use cases.","The two new powerpcspe signatures can be reused by disassembler and firmware-analysis tools to distinguish PowerPC from PowerPC SPE code."],"supporting_citations":[{"why":"Supplies the byte-frequency and endianness-signature method this paper reconstructs and compares against.","marker":"[10]"},{"why":"Supplies the extended-signature variant and the complete-binary test protocol that this paper replicates and improves on.","marker":"[15]"},{"why":"Introduces byte-frequency content-based file type detection, the conceptual foundation of the feature set.","marker":"[35]"},{"why":"Provides the function prologue and epilogue byte signatures used as 31 of the 293 features.","marker":"[47]"},{"why":"Demonstrates random-forest firmware classification, informing the classifier choice and the firmware motivation.","marker":"[13]"},{"why":"Quantifies the share of IoT firmware analysis failures caused by wrong ISA identification, motivating the problem.","marker":"[11]"}],"fun_headline_variants":["Whole-binary classifier hits 99.2% CPU architecture accuracy","Train on full executables: 99.2% ISA detection, beats code sections","99.2% architecture ID from entire binary files, not just code","Full-binary random forest achieves 99.2% architecture accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 99.2% result assumes that testing on complete executable files, whose headers openly record the target architecture, tells us how the classifier would perform on the headerless and fragmented object code the paper is ultimately about.","fun_headline_variants_meta":{"raw":{"variants":["Whole-binary classifier hits 99.2% CPU architecture accuracy","Train on full executables: 99.2% ISA detection, beats code sections","99.2% architecture ID from entire binary files, not just code","Full-binary random forest achieves 99.2% architecture accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000279,"raw_usage":{"total_tokens":1683,"prompt_tokens":995,"completion_tokens":688,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":611,"completion_tokens_details":{"reasoning_tokens":608}},"tokens_in":611,"tokens_out":688,"duration_ms":6374,"temperature":1.0,"reasoning_tokens":608,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:13:37.990536+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Zero out or remove the architecture field from the headers of the complete-binary test set and repeat the 99.2% experiment; if accuracy falls sharply, the classifier was using header metadata rather than code bytes.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the byte-frequency and endianness-signature method this paper reconstructs and compares against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the extended-signature variant and the complete-binary test protocol that this paper replicates and improves on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces byte-frequency content-based file type detection, the conceptual foundation of the feature set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates random-forest firmware classification, informing the classifier choice and the firmware motivation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Quantifies the share of IoT firmware analysis failures caused by wrong ISA identification, motivating the problem."}],"review_version":1}