{"id":"77b757b5-6825-4a27-aa97-b556cad3fe3e","arxiv_id":"2412.07538","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A neural-decompilation-to-classification pipeline reports 95% F1 binary and 83% multi-class vulnerability prediction on Juliet, with comparison and split caveats.","lead":"This paper tests whether translating binary executables back into C/C++ source with neural networks, then running standard vulnerability classifiers on that reconstructed source, can spot security flaws without access to the original code. On the synthetic Juliet benchmark the pipeline hits 95% F1 for vulnerable/non-vulnerable and 83% accuracy for vulnerability type, but the evaluation has unresolved data-split and comparison issues.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central comparative claim rests on a confounded cross-paper comparison: no same-pipeline baseline on assembly or IR, so the 7%/6% improvement cannot be attributed to decompiled source.","rationale":"I read the paper's central claim as comparative: neural decompilation as an intermediate representation improves vulnerability prediction over assembly/IR. This requires an experiment where the only varying factor is the representation, with model, data, and split held fixed. The paper instead compares its best transformer on decompiled code to Schaad et al.'s RNN on LLVM IR across different datasets and CWE selections. That is a confounded cross-paper comparison. I did not find an internal assembly/IR baseline anywhere in Section 4.2, and the qualitative decompilation example does not speak to downstream classification. The reader's split concern is a plausible data-integrity issue, and the full-text wording '75,149 C/C++ functions for both training and testing' is genuinely ambiguous, but the controlled-baseline gap is the more load-bearing issue because it affects the central claim regardless of split. I would require a same-pipeline ablation before accepting the 7%/6% improvement claim; with that addition the paper could become acceptable, so I leave the reader's CONDITIONAL verdict unchanged.","tokens_in":10069,"tokens_out":5059,"duration_ms":50869,"concrete_test":"Run a controlled ablation on the same 80/20 split used in Section 4.2: take the same Juliet functions and produce three inputs — (1) raw objdump assembly tokens, (2) LLVM IR as in Schaad et al., and (3) fairseq-decompiled source — then train the same CodeGPT and CodeBERT classifiers on the original source train split and evaluate on each representation's test split. Report accuracy and F1 per representation. If decompiled source does not beat assembly and LLVM IR by a statistically significant margin on identical data and models, the 7%/6% improvement over prior IR-based work is not evidence for the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline claim is that decompiled source outperforms disassembled or IR-based representations, with 95% F1 (bi-class) and 83% accuracy (multi-class), a 7%/6% gain over Schaad et al. Section 4.2 (Table 4) supports this only by comparing CodeGPT/CodeBERT on neurally decompiled C/C++ with an SRNN on LLVM IR from a different paper, and the table note admits the multi-class CWE sets differ (20 manually chosen vs 23 in [9]). Because the two arms differ in model architecture, training data, preprocessing, label set, and evaluation split, the observed differences cannot be attributed to the intermediate representation. The paper even calls [9] 'disassembled code' in the introduction, while Table 5 lists its representation as LLVM IR. No experiment in the paper feeds the same classifiers with assembly tokens or LLVM IR on the same split, so the central representation-advantage claim is not tested. The train/test split ambiguity flagged by the reader is important but secondary: even a clean split would not fix the missing within-paper baseline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a binary-code vulnerability prediction pipeline: compile Juliet C/C++ test cases to x86-64 binaries, disassemble with objdump, preprocess functions, translate assembly to C/C++ with a fine-tuned neural decompiler (fairseq), and then apply deep-learning classifiers (CodeGPT, CodeBERT, CodeT5+, SRNN, LSTM, GRU) to the decompiled code for binary vulnerable/benign prediction and multi-class CWE prediction. The authors report a 59% edit-distance score for decompilation and 95% F1 / 83% accuracy for the two classification tasks, and claim a 7%/6% improvement over Schaad et al.'s LLVM-IR-based approach.","tokens_in":10231,"tokens_out":6035,"duration_ms":50255,"significance":"If the representation advantage were established, the pipeline would be a practical step toward vulnerability prediction on binaries without source code, using only standard deep-learning components. Strengths include the use of the standard Juliet suite, evaluation of multiple model families, inclusion of a Ghidra baseline, and a SHAP-based sanity check on influential features. However, the paper's central comparative claim is not yet supported because the comparison to Schaad et al. changes both the model and the input representation, and because the train/test separation of the decompiler is not clearly reported. The reported gains are therefore not attributable to decompilation on the evidence presented.","major_comments":[{"comment":"The headline claim that decompiled source outperforms disassembled/IR-based representations rests on a comparison to Schaad et al. in which both the model (SRNN vs CodeGPT/CodeBERT) and the intermediate representation (LLVM IR vs decompiled C/C++) change, and in which the multi-class label sets also differ (23 vs 20 CWEs). This is a confounded comparison: the 7% bi-class and 6% multi-class differences cannot be attributed to the decompiled representation. Notably, the authors' own SRNN on decompiled C/C++ (Table 4) achieves 88% F1, exactly matching Schaad et al.'s SRNN on LLVM IR (88% F1), which suggests the improvement comes from the transformer models rather than from decompilation. Add within-paper baselines that feed the same classifiers with assembly tokens or LLVM IR on the same split to isolate the representation effect, or restrict the claim to a model-plus-representation comparison.","section":"Section 4.2, Table 4"},{"comment":"Table 2 and Section 4.1 report that fairseq was trained and tested on '75,149 C/C++ functions for both training and testing' without specifying the split. If the decompiler was trained and evaluated on the same functions, the 59% ED and the downstream classification accuracy could reflect memorization rather than generalization. Specify the exact split (e.g., 80/20 stratified), confirm that the vulnerability-prediction test set is disjoint from the decompiler training set, and report whether any validation set was used for early stopping or model selection.","section":"Section 4.1, Table 2"},{"comment":"Section 3.2 states that models are trained on original C/C++ programs and tested on decompiled programs, whereas Section 4.2 says the decompiled code is split 80-20 for classification. These two statements are inconsistent and matter for the 'binary-only' claim: if the classifiers are trained on original source, the pipeline requires source at training time; if they are trained on decompiled code, the distribution shift is different. Clarify which setup was used and describe the split for each classification task.","section":"Section 3.2 vs Section 4.2"},{"comment":"The multi-class comparison is weakened by the manual selection of 20 CWE classes chosen for 'distinctness' and Top-25 status. Because the selection is made by the authors and differs from Schaad et al.'s 23 classes, the 6% accuracy gain may reflect a difference in task difficulty rather than a representation advantage. Justify the selection with pre-registered criteria or report results on the same CWE set as Schaad et al., and include per-class results to check for cherry-picking.","section":"Section 4.2, multi-class classification"},{"comment":"The decompilation comparison to Katz et al. and Hosseini et al. is not quantitative, because those works use a complementary edit-distance definition (as the paper itself states) plus different training data, architectures, and maximum target lengths. The statement that the proposed approach 'outperforms the state-of-the-art' in decompilation is therefore unsupported; report metrics computed with an identical evaluation protocol, or temper the claim to 'competitive' rather than 'outperforming'.","section":"Table 2, Section 4.1"}],"minor_comments":[{"comment":"The abstract and introduction state that CodeBERT achieves an 83% F1-score for multi-class classification, but Table 4 reports an 82% F1-score and 83% accuracy; please correct the inconsistency.","section":"Abstract and Section 1"},{"comment":"The reference to the automated build approach by Richardson et al. appears as '[?]' in the text; the citation is unresolved and should be completed.","section":"Section 3"},{"comment":"The phrase 'we applied toss reduction on tokenized code via BPE' appears to be a typo; clarify whether this means tail reduction, outlier removal, or another truncation procedure.","section":"Section 3"},{"comment":"The introduction refers to Schaad et al. as using 'disassembled code', while Table 5 lists their representation as LLVM IR; please use consistent terminology for disassembly and intermediate representations.","section":"Introduction and Table 5"},{"comment":"The experimental results are reported for a single split without confidence intervals or significance tests; adding repeated runs or statistical testing would increase confidence in the observed differences among models.","section":"Section 4.2"},{"comment":"The Flawfinder comparison on the multi-class task uses only approximately 500 programs; this small subset makes the 'equivalent to random' statement difficult to interpret, and the size should be acknowledged in the discussion.","section":"Section 4.2, Flawfinder comparison"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a security/systems workshop, but the unresolved citation and lack of artifacts limit reproducibility. The central comparative claim should be reframed as a model-plus-representation comparison, and the decompiler split must be reported explicitly. The strongest internal evidence against the representation-only claim is that the authors' own SRNN on decompiled code matches Schaad et al.'s SRNN on LLVM IR, so the paper's conclusion overreaches its data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is the first to actually run neural decompilation followed by DL vulnerability prediction on binaries, and the raw numbers are impressive. That part is real. But the headline claim — that decompiled source beats disassembly or IR as an intermediate representation — rests on a cross-paper comparison that changes model, representation, and label set all at once. No experiment in the paper feeds the same classifiers assembly or IR on the same split. So the 7%/6% improvement is not actually demonstrated.\n\nWhat is new and good: they build a full pipeline on 75k Juliet functions, train fairseq from scratch, and compare it to CodeBERT/CodeT5+ decoders; they report several quality metrics; they test six downstream classifiers; they include Ghidra and Flawfinder as baselines; they even do a SHAP analysis to check for Clever Hans behavior. The observation that pre-trained models underperform the from-scratch fairseq on decompilation is interesting and worth exploring. The qualitative example in Table 3 is helpful.\n\nThe soft spots are serious, though. First, as noted, the comparison to Schaad et al. changes the architecture (SRNN vs Transformer), the representation (LLVM IR vs decompiled C), and the class set (23 CWEs vs 20 manually chosen). The paper itself admits the class sets differ in the Table 4 footnote, yet the conclusion still says the improvement comes from using source code as IR. That is not a controlled comparison.\n\nSecond, Section 4.1 says fairseq was trained and tested on 75,149 functions without reporting a split. Later, Section 4.2 says the classification data was split 80/20 with stratification, but that seems to refer to the classification task after decompilation. It is unclear whether the decompiler's training set included the functions used for the decompiled test set. That ambiguity matters: if the decompiler memorized functions, the downstream accuracy could partly reflect memorization. The reader's concern here is legitimate, and the paper needs to state the split explicitly.\n\nThird, the multi-class setting uses 20 manually selected CWEs, and Table 4's comparison to Schaad is therefore apples-to-oranges. The claim of 83% vs 77% is not a fair head-to-head.\n\nOn balance: the pipeline is plausible and the work is honest in tone — limitations of neural decompilation are acknowledged. But the evaluation protocol needs a same-pipeline baseline (same classifiers on assembly tokens and/or LLVM IR, same split, same class set) before the central advantage claim can be accepted. If the authors add that baseline and clarify the decompiler split, the paper would be a solid contribution to binary-only vulnerability prediction.\n\nMy recommendation: send it to peer review with the expectation of heavy revision. The idea deserves referee time, but the current comparison is not sufficient. I would not cite it in its present form.","headline":"A genuinely new pipeline with strong numbers, but the central comparative claim is not tested within the paper.","tokens_in":10790,"tokens_out":2216,"would_cite":false,"duration_ms":20854,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Neural decompilation of binaries into C/C++ source enables deep-learning vulnerability prediction at 95% F1 and 83% accuracy, outperforming IR-based baselines on the Juliet benchmark.","keywords":["neural decompilation","vulnerability prediction","binary code analysis","deep learning","CWE classification","transformer","Juliet test suite","neural machine translation"],"falsifier":"Re-run the pipeline with a strict, documented partition: train the fairseq decompiler on one disjoint subset of Juliet functions and evaluate the vulnerability classifiers on decompiled output from a held-out subset; if normalized edit similarity falls well below 59% or the classification F1 drops toward the 88% disassembly baseline, the central claim of generalizable, decompilation-assisted prediction would be refuted.","tokens_in":9833,"feed_emoji":"🛡️","tokens_out":11393,"duration_ms":89710,"temperature":0.7,"pith_summary":"Vulnerability prediction normally assumes source code is available, which rules out most closed-source and legacy software. This paper tests a bridge: disassemble the binary into assembly, translate that assembly back into C/C++ with a transformer-based neural decompiler, and then run deep-learning classifiers on the recovered source. On the synthetic Juliet benchmark, the authors report 95% F1 for vulnerable/non-vulnerable classification and 83% accuracy for a 20-class CWE-type task, beating an LLVM-IR-based prior work by 7 and 6 percentage points respectively. The study's broader claim is that treating binary code as plain text and recovering source-like structure is enough to make source-level vulnerability prediction work on binaries.","feed_headline":"Decompiling binaries to C/C++ lifts vuln prediction to 95% F1","feed_subtitle":"Classifying decompiled source beats IR-based binary analysis by 7 and 6 points on the Juliet benchmark.","key_machinery":"The central mechanism is neural decompilation cast as neural machine translation: the binary is disassembled with objdump, reduced to the .text section with library functions, OS directives, and main() removed, function names replaced by placeholders, and tokenized with BPE; an encoder-decoder transformer then generates C/C++ source from that token stream. This recovered source becomes the input to standard deep-learning vulnerability classifiers. The paper's key design choice is that all code is treated as plain text, so no control-flow graphs or intermediate representations need to be constructed; the transformer is expected to learn program structure implicitly, and the authors argue this is why decompiled source carries more vulnerability-relevant signal than assembly or IR.","core_discovery":"The authors set out to show that neural decompilation can supply an intermediate representation for vulnerability prediction that is better than disassembly or hand-built IRs. Their evidence is a pipeline in which objdump disassembly is preprocessed into token sequences and translated to C/C++ by an encoder-decoder transformer (fairseq is the best, reaching 59% normalized edit similarity), and the resulting source is fed to classifiers such as CodeGPT and CodeBERT. The reported outcome is 95% F1 on the binary vulnerable/non-vulnerable task and 83% accuracy with 82% F1 on the 20-class CWE task, improvements of 7 and 6 percentage points over the LLVM-IR method of Schaad et al. The paper concludes that decompiled source code, despite imperfect edit similarity, preserves enough semantic structure to outperform representations that require explicit control-flow and data-flow analysis.","pith_inferences":["If the reported gains carry beyond the synthetic Juliet corpus, this method would give security teams a path to classify vulnerabilities in firmware and proprietary binaries using source-level models, but real-world binaries with inlining and obfuscation are an untested regime.","The paper leaves implicit that training downstream classifiers directly on decompiled source (instead of original source) could make the pipeline entirely source-free and possibly further close the gap; this can be tested empirically.","Because both the decompiler and the classifiers work on the same corpus, a disjoint split is the first external check; without it, the reported edit similarity and F1 scores cannot be distinguished from memorization."],"forward_implications":["Binary-only security assessments of proprietary, legacy, or firmware software could adopt source-code-trained deep-learning models and standardized CWE taxonomies, without needing source access or custom binary feature engineering.","The same decompile-then-classify pipeline could be retargeted to other architectures or languages by retraining the transformer on new assembly/source pairs, rather than writing new heuristic decompiler rules.","Because a from-scratch transformer on 75k functions beat pre-trained code models for decompilation, large curated function-level corpora may be more valuable than massive generic pre-training for this translation task, according to the paper's interpretation.","If the comparison to Schaad et al. holds, decompiled source as an intermediate representation generalizes better than LLVM IR for vulnerability prediction, suggesting that information loss in IR construction may be the bottleneck."],"supporting_citations":[{"why":"The LLVM-IR-based binary vulnerability prediction baseline (88% F1, 77% accuracy) that the proposed decompiled-source approach claims to surpass by 7 and 6 points.","marker":"[9]"},{"why":"The compilable, CWE-labeled Juliet dataset that supplies all 75,149 functions and the classification labels for both tasks.","marker":"[18]"},{"why":"The 2,000,000-function neural decompilation study (54% edit similarity) used as the main decompilation quality comparison.","marker":"[29]"},{"why":"The earlier RNN decompiler (30% edit similarity on 700,000 snippets) providing a second neural decompilation benchmark.","marker":"[36]"},{"why":"Ghidra, the traditional decompiler whose 23% edit similarity and less readable output are contrasted with neural decompilation.","marker":"[4]"},{"why":"Flawfinder, the static-analysis baseline that scores near random and motivates the neural classification design.","marker":"[41]"}],"fun_headline_variants":["Neural decompilation lifts binary vuln prediction to 95% F1","Decompiled source beats IR for binary vulnerability prediction","Neural decompilation improves binary vuln classification by 7 points","From binary to C: neural decompilation boosts vuln detection","Binary vuln prediction gains 7 points via neural decompilation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results are only meaningful if the functions used to test the vulnerability classifiers were not among those used to train the neural decompiler; the paper does not explicitly report this split, and if the same 75,149 functions were used for both training and testing, the reported accuracy could reflect memorization rather than genuine decompilation ability.","fun_headline_variants_meta":{"raw":{"variants":["Neural decompilation lifts binary vuln prediction to 95% F1","Decompiled source beats IR for binary vulnerability prediction","Neural decompilation improves binary vuln classification by 7 points","From binary to C: neural decompilation boosts vuln detection","Binary vuln prediction gains 7 points via neural decompilation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1266,"prompt_tokens":828,"completion_tokens":438,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":444,"completion_tokens_details":{"reasoning_tokens":350}},"tokens_in":444,"tokens_out":438,"duration_ms":3760,"temperature":1.0,"reasoning_tokens":350,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:44:35.257525+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline with a strict, documented partition: train the fairseq decompiler on one disjoint subset of Juliet functions and evaluate the vulnerability classifiers on decompiled output from a held-out subset; if normalized edit similarity falls well below 59% or the classification F1 drops toward the 88% disassembly baseline, the central claim of generalizable, decompilation-assisted prediction would be refuted.","supporting_citations":[{"cited_title":"Deep-learning-based vulnerability detection in binary executables,","cited_arxiv_id":null,"evidence_quote":"The LLVM-IR-based binary vulnerability prediction baseline (88% F1, 77% accuracy) that the proposed decompiled-source approach claims to surpass by 7 and 6 points."},{"cited_title":"Juliet Test Suite C/C++ 1.3,","cited_arxiv_id":null,"evidence_quote":"The compilable, CWE-labeled Juliet dataset that supplies all 75,149 functions and the classification labels for both tasks."},{"cited_title":"Using recurrent neural networks for decompilation,","cited_arxiv_id":null,"evidence_quote":"The earlier RNN decompiler (30% edit similarity on 700,000 snippets) providing a second neural decompilation benchmark."},{"cited_title":"Ghidra Software Reverse Engineering Framework,","cited_arxiv_id":null,"evidence_quote":"Ghidra, the traditional decompiler whose 23% edit similarity and less readable output are contrasted with neural decompilation."},{"cited_title":"Flawfinder: a static analysis tool for finding vulnera- bilities in c/c++ source code,","cited_arxiv_id":null,"evidence_quote":"Flawfinder, the static-analysis baseline that scores near random and motivates the neural classification design."}],"review_version":1}