REVIEW 5 major objections 6 minor 48 references
Can Neural Decompilation Assist Vulnerability Prediction on Binary Code?
T0 review · 5 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Neural decompilation of binaries into C/C++ source enables deep-learning vulnerability prediction at 95% F1 and 83% accuracy, outperforming IR-based baselines on the Juliet benchmark.
desk verdict A genuinely new pipeline with strong numbers, but the central comparative claim is not tested within the paper. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is neural decompilation cast as neural machine translation: the binary is disassembled with objdump, reduced to the .text section with library functions, OS directives, and main() removed, function names replaced by placeholders, and tokenized with BPE; an encoder-decoder transformer then generates C/C++ source from that token stream. This recovered source becomes the input to standard deep-learning vulnerability classifiers. The paper's key design choice is that all code is treated as plain text, so no control-flow graphs or intermediate representations need to be constructed; the transformer is expected to learn program structure implicitly, and the authors argue this is why decompiled source carries more vulnerability-relevant signal than assembly or IR.
What would settle it
Re-run the pipeline with a strict, documented partition: train the fairseq decompiler on one disjoint subset of Juliet functions and evaluate the vulnerability classifiers on decompiled output from a held-out subset; if normalized edit similarity falls well below 59% or the classification F1 drops toward the 88% disassembly baseline, the central claim of generalizable, decompilation-assisted prediction would be refuted.
Extended reading notes
Core claim
The authors set out to show that neural decompilation can supply an intermediate representation for vulnerability prediction that is better than disassembly or hand-built IRs. Their evidence is a pipeline in which objdump disassembly is preprocessed into token sequences and translated to C/C++ by an encoder-decoder transformer (fairseq is the best, reaching 59% normalized edit similarity), and the resulting source is fed to classifiers such as CodeGPT and CodeBERT. The reported outcome is 95% F1 on the binary vulnerable/non-vulnerable task and 83% accuracy with 82% F1 on the 20-class CWE task, improvements of 7 and 6 percentage points over the LLVM-IR method of Schaad et al. The paper concludes that decompiled source code, despite imperfect edit similarity, preserves enough semantic structure to outperform representations that require explicit control-flow and data-flow analysis.
Load-bearing premise
The results are only meaningful if the functions used to test the vulnerability classifiers were not among those used to train the neural decompiler; the paper does not explicitly report this split, and if the same 75,149 functions were used for both training and testing, the reported accuracy could reflect memorization rather than genuine decompilation ability.
Editorial extensions
If this is right
- Binary-only security assessments of proprietary, legacy, or firmware software could adopt source-code-trained deep-learning models and standardized CWE taxonomies, without needing source access or custom binary feature engineering.
- The same decompile-then-classify pipeline could be retargeted to other architectures or languages by retraining the transformer on new assembly/source pairs, rather than writing new heuristic decompiler rules.
- Because a from-scratch transformer on 75k functions beat pre-trained code models for decompilation, large curated function-level corpora may be more valuable than massive generic pre-training for this translation task, according to the paper's interpretation.
- If the comparison to Schaad et al. holds, decompiled source as an intermediate representation generalizes better than LLVM IR for vulnerability prediction, suggesting that information loss in IR construction may be the bottleneck.
Reading between the lines
- If the reported gains carry beyond the synthetic Juliet corpus, this method would give security teams a path to classify vulnerabilities in firmware and proprietary binaries using source-level models, but real-world binaries with inlining and obfuscation are an untested regime.
- The paper leaves implicit that training downstream classifiers directly on decompiled source (instead of original source) could make the pipeline entirely source-free and possibly further close the gap; this can be tested empirically.
- Because both the decompiler and the classifiers work on the same corpus, a disjoint split is the first external check; without it, the reported edit similarity and F1 scores cannot be distinguished from memorization.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a binary-code vulnerability prediction pipeline: compile Juliet C/C++ test cases to x86-64 binaries, disassemble with objdump, preprocess functions, translate assembly to C/C++ with a fine-tuned neural decompiler (fairseq), and then apply deep-learning classifiers (CodeGPT, CodeBERT, CodeT5+, SRNN, LSTM, GRU) to the decompiled code for binary vulnerable/benign prediction and multi-class CWE prediction. The authors report a 59% edit-distance score for decompilation and 95% F1 / 83% accuracy for the two classification tasks, and claim a 7%/6% improvement over Schaad et al.'s LLVM-IR-based approach.
Significance. If the representation advantage were established, the pipeline would be a practical step toward vulnerability prediction on binaries without source code, using only standard deep-learning components. Strengths include the use of the standard Juliet suite, evaluation of multiple model families, inclusion of a Ghidra baseline, and a SHAP-based sanity check on influential features. However, the paper's central comparative claim is not yet supported because the comparison to Schaad et al. changes both the model and the input representation, and because the train/test separation of the decompiler is not clearly reported. The reported gains are therefore not attributable to decompilation on the evidence presented.
major comments (5)
- [Section 4.2, Table 4] The headline claim that decompiled source outperforms disassembled/IR-based representations rests on a comparison to Schaad et al. in which both the model (SRNN vs CodeGPT/CodeBERT) and the intermediate representation (LLVM IR vs decompiled C/C++) change, and in which the multi-class label sets also differ (23 vs 20 CWEs). This is a confounded comparison: the 7% bi-class and 6% multi-class differences cannot be attributed to the decompiled representation. Notably, the authors' own SRNN on decompiled C/C++ (Table 4) achieves 88% F1, exactly matching Schaad et al.'s SRNN on LLVM IR (88% F1), which suggests the improvement comes from the transformer models rather than from decompilation. Add within-paper baselines that feed the same classifiers with assembly tokens or LLVM IR on the same split to isolate the representation effect, or restrict the claim to a model-plus-representation comparison.
- [Section 4.1, Table 2] Table 2 and Section 4.1 report that fairseq was trained and tested on '75,149 C/C++ functions for both training and testing' without specifying the split. If the decompiler was trained and evaluated on the same functions, the 59% ED and the downstream classification accuracy could reflect memorization rather than generalization. Specify the exact split (e.g., 80/20 stratified), confirm that the vulnerability-prediction test set is disjoint from the decompiler training set, and report whether any validation set was used for early stopping or model selection.
- [Section 3.2 vs Section 4.2] Section 3.2 states that models are trained on original C/C++ programs and tested on decompiled programs, whereas Section 4.2 says the decompiled code is split 80-20 for classification. These two statements are inconsistent and matter for the 'binary-only' claim: if the classifiers are trained on original source, the pipeline requires source at training time; if they are trained on decompiled code, the distribution shift is different. Clarify which setup was used and describe the split for each classification task.
- [Section 4.2, multi-class classification] The multi-class comparison is weakened by the manual selection of 20 CWE classes chosen for 'distinctness' and Top-25 status. Because the selection is made by the authors and differs from Schaad et al.'s 23 classes, the 6% accuracy gain may reflect a difference in task difficulty rather than a representation advantage. Justify the selection with pre-registered criteria or report results on the same CWE set as Schaad et al., and include per-class results to check for cherry-picking.
- [Table 2, Section 4.1] The decompilation comparison to Katz et al. and Hosseini et al. is not quantitative, because those works use a complementary edit-distance definition (as the paper itself states) plus different training data, architectures, and maximum target lengths. The statement that the proposed approach 'outperforms the state-of-the-art' in decompilation is therefore unsupported; report metrics computed with an identical evaluation protocol, or temper the claim to 'competitive' rather than 'outperforming'.
minor comments (6)
- [Abstract and Section 1] The abstract and introduction state that CodeBERT achieves an 83% F1-score for multi-class classification, but Table 4 reports an 82% F1-score and 83% accuracy; please correct the inconsistency.
- [Section 3] The reference to the automated build approach by Richardson et al. appears as '[?]' in the text; the citation is unresolved and should be completed.
- [Section 3] The phrase 'we applied toss reduction on tokenized code via BPE' appears to be a typo; clarify whether this means tail reduction, outlier removal, or another truncation procedure.
- [Introduction and Table 5] The introduction refers to Schaad et al. as using 'disassembled code', while Table 5 lists their representation as LLVM IR; please use consistent terminology for disassembly and intermediate representations.
- [Section 4.2] The experimental results are reported for a single split without confidence intervals or significance tests; adding repeated runs or statistical testing would increase confidence in the observed differences among models.
- [Section 4.2, Flawfinder comparison] The Flawfinder comparison on the multi-class task uses only approximately 500 programs; this small subset makes the 'equivalent to random' statement difficult to interpret, and the size should be acknowledged in the discussion.
Circularity Check
No significant circularity: the pipeline is empirical and not self-deriving; the main weaknesses are a confounded cross-paper baseline and an ambiguous train/test split, neither of which exhibits a definitional reduction.
full rationale
The paper's derivation chain is empirical rather than mathematical: binaries are compiled, disassembled to assembly, neurally decompiled to C/C++, and the decompiled functions are fed to classifiers that were trained on original source code. No fitted parameter is later renamed a prediction: the decompiler and the vulnerability classifiers are separate models with separate objectives, and the vulnerability labels come from the Juliet dataset. The only overlapping-author citation is reference [39], used to justify a complementary Edit Distance variant; this is not load-bearing for the central claim and therefore does not constitute circularity under the stated rules. The self-reported gains over prior work are weakened by a cross-paper comparison: Table 4 contrasts CodeGPT/CodeBERT on decompiled C/C++ with an SRNN on LLVM IR from Schaad et al. [9], with different model architectures, preprocessing, and CWE sets, as the table note itself concedes. That is a benchmarking confound, not a circular reduction. The phrase 'using 75,149 C/C++ functions for both training and testing' (Section 4.1) is ambiguous and could indicate that decompilation quality was measured without a clear held-out split, which would be a data-leakage risk for the downstream accuracy numbers; however, the paper does not explicitly state that the same functions were used to fit the decompiler and to evaluate it, so this remains an evaluation-protocol concern rather than an exhibitable case of a prediction reducing to its own input. On the evidence quotable from the paper, there is no self-definitional equation, no fitted-input-called-prediction step, and no load-bearing self-citation chain. The appropriate circularity score is therefore low, reflecting only a minor non-load-bearing self-citation and an ambiguous evaluation statement, not a circular derivation.
Assumptions & free parameters
free parameters (2)
- Token-length percentile thresholds =
5th and 95th percentiles
- Selected 20 CWE classes =
190, 762, 121, 590, 194, 197, 78, 36, 23, 690, 369, 457, 134, 680, 758, 400, 789, 253, 398, 761
assumptions (4)
- domain assumption Assembly code extracted from the .text section preserves vulnerability-relevant logic after removing libraries, OS directives, and main.
- domain assumption Juliet C/C++ 1.3 test cases are representative of real-world vulnerable and non-vulnerable functions.
- domain assumption The classifier trained on original source transfers to decompiled source without retraining.
- domain assumption Training and test splits are disjoint for both the decompiler and the classifiers.
Cite this review
Pith. "Pith review of Can Neural Decompilation Assist Vulnerability Prediction on Binary Code?." pith.science (2026). https://pith.science/paper/67FOXHHX
@misc{pith2026241207538,
author = {Pith},
title = {Pith review of: Can Neural Decompilation Assist Vulnerability Prediction on Binary Code?},
year = {2026},
howpublished = {\url{https://pith.science/paper/67FOXHHX}},
note = {Machine review of arXiv:2412.07538}
}
read the original abstract
Vulnerability prediction is valuable in identifying security issues efficiently, even though it requires the source code of the target software system, which is a restrictive hypothesis. This paper presents an experimental study to predict vulnerabilities in binary code without source code or complex representations of the binary, leveraging the pivotal idea of decompiling the binary file through neural decompilation and predicting vulnerabilities through deep learning on the decompiled source code. The results outperform the state-of-the-art in both neural decompilation and vulnerability prediction, showing that it is possible to identify vulnerable programs with this approach concerning bi-class (vulnerable/non-vulnerable) and multi-class (type of vulnerability) analysis.
Figures
Reference graph
Works this paper leans on
-
[1]
Han-bsvd: a hierarchical attention network for binary software vulnerability detection,
H. Yan, S. Luo, L. Pan, and Y. Zhang, “Han-bsvd: a hierarchical attention network for binary software vulnerability detection, ” Computers & Security, 2021
work page 2021
-
[2]
Generalized vulnerability extrapolation using abstract syntax trees,
F. Yamaguchi, M. Lottmann, and K. Rieck, “Generalized vulnerability extrapolation using abstract syntax trees, ” inProceedings of the 28th annual computer security applications conference , 2012
work page 2012
-
[3]
Vulnerability detection for source code using contextual lstm,
A. Xu, T. Dai, H. Chen, Z. Ming, and W. Li, “Vulnerability detection for source code using contextual lstm, ” in2018 5th international conference on systems and informatics (ICSAI) . IEEE, 2018
work page 2018
-
[4]
Ghidra Software Reverse Engineering Framework,
N. S. Agency, “Ghidra Software Reverse Engineering Framework, ” ht tps://github.com/NationalSecurityAgency/ghidra
-
[5]
Stack-based Buffer Overflow Detection using Recurrent Neural Networks
W. A. Dahl, L. Erdodi, and F. M. Zennaro, “Stack-based buffer over- flow detection using recurrent neural networks, ” arXiv preprint arXiv:2012.15116, 2020
work page Pith review arXiv 2012
-
[6]
discovre: Efficient cross-architecture identification of bugs in binary code
S. Eschweiler, K. Yakdan, E. Gerhards-Padilla et al., “discovre: Efficient cross-architecture identification of bugs in binary code. ” inNdss, 2016
work page 2016
-
[7]
Toward large-scale vulnerability discovery using machine learning,
G. Grieco, G. L. Grinblat, L. Uzal, S. Rawat, J. Feist, and L. Mounier, “Toward large-scale vulnerability discovery using machine learning, ” in Proceedings of the Sixth ACM Conference on Data and Application Security and Privacy, 2016
work page 2016
-
[8]
Cross- architecture bug search in binary executables,
J. Pewny, B. Garmany, R. Gawlik, C. Rossow, and T. Holz, “Cross- architecture bug search in binary executables, ” in2015 IEEE Symposium on Security and Privacy . IEEE, 2015
work page 2015
Show all 48 references
-
[9]
Deep-learning-based vulnerability detection in binary executables,
A. Schaad and D. Binder, “Deep-learning-based vulnerability detection in binary executables, ” inFoundations and Practice of Security: 15th International Symposium, FPS 2022, Ottawa, ON, Canada, December 12–14, 2022, Revised Selected Papers . Springer, 2023
2022
-
[10]
Recurrent neural network based binary code vulnerability detection,
J. Zheng, J. Pang, X. Zhang, X. Zhou, M. Li, and J. Wang, “Recurrent neural network based binary code vulnerability detection, ” inProceed- ings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence, 2019
2019
-
[11]
Lifting assembly to intermediate repre- sentation: A novel approach leveraging compilers,
N. Hasabnis and R. Sekar, “Lifting assembly to intermediate repre- sentation: A novel approach leveraging compilers, ” inProceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems , 2016, pp. 311–324
2016
-
[12]
Bin2vec: learning representations of binary executable programs for security tasks,
S. Arakelyan, S. Arasteh, C. Hauser, E. Kline, and A. Galstyan, “Bin2vec: learning representations of binary executable programs for security tasks, ”Cybersecurity, 2021
2021
-
[13]
Common weakness enumeration (cwe),
T. M. Corporation, “Common weakness enumeration (cwe), ” https: //cwe.mitre.org/
-
[14]
The MITRE Corporation,
The MITRE Corporation, “The MITRE Corporation, ” https://www.mi tre.org/
-
[15]
RetDec,
Avast, “RetDec, ” https://github.com/avast/retdec
-
[16]
Deep learning based vulnerability detection: Are we there yet?
S. Chakraborty, R. Krishna, Y. Ding, and B. Ray, “Deep learning based vulnerability detection: Are we there yet?” IEEE Transactions on Soft- ware Engineering, 2022
2022
-
[17]
Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks,
Y. Zhou, S. Liu, J. Siow, X. Du, and Y. Liu, “Devign: Effective vulner- ability identification by learning comprehensive program semantics via graph neural networks, ” 2019
2019
-
[18]
Juliet Test Suite C/C++ 1.3,
NIST, “Juliet Test Suite C/C++ 1.3, ” https://samate.nist.gov/SARD/test- suites/112
-
[19]
A c/c++ code vulnerability dataset with code changes and cve summaries
J. Fan, Y. Li, S. Wang, and T. N. Nguyen, “A c/c++ code vulnerability dataset with code changes and cve summaries. ” New York, NY, USA: Association for Computing Machinery, 2020. [Online]. Available: https://doi.org/10.1145/3379597.3387501
2020
-
[20]
Diversevul: A new vulnerable source code dataset for deep learning based vulnera- bility detection,
Y. Chen, Z. Ding, L. Alowain, X. Chen, and D. Wagner, “Diversevul: A new vulnerable source code dataset for deep learning based vulnera- bility detection, ” 2023
2023
-
[21]
Software Assurance Reference Dataset,
NIST, “Software Assurance Reference Dataset, ” https://samate.nist.go v/SARD/
-
[22]
National Institute of Standards and Technology,
NIST, “National Institute of Standards and Technology, ” https://www. nist.gov/
-
[23]
Vuldeelocator: a deep learning-based fine-grained vulnerability detector,
Z. Li, D. Zou, S. Xu, Z. Chen, Y. Zhu, and H. Jin, “Vuldeelocator: a deep learning-based fine-grained vulnerability detector, ”IEEE Transactions on Dependable and Secure Computing , 2021
2021
-
[24]
Vuldeepecker: A deep learning-based system for vulnerability detec- tion,
Z. Li, D. Zou, S. Xu, X. Ou, H. Jin, S. Wang, Z. Deng, and Y. Zhong, “Vuldeepecker: A deep learning-based system for vulnerability detec- tion, ”arXiv preprint arXiv:1801.01681, 2018
2018 arXiv
-
[25]
A code centric evaluation of c/c++ vulnerability datasets for deep learning based vul- nerability detection techniques,
R. Jain, N. Gervasoni, M. Ndhlovu, and S. Rawat, “A code centric evaluation of c/c++ vulnerability datasets for deep learning based vul- nerability detection techniques, ” inProceedings of the 16th Innovations in Software Engineering Conference , 2023
2023
-
[26]
A comprehensive detection of memory corruption vulnerabilities for c/c++ programs,
Y. Gao, L. Chen, G. Shi, and F. Zhang, “A comprehensive detection of memory corruption vulnerabilities for c/c++ programs, ” in2018 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Ubiq- uitous Computing & Communications, Big Data & Cloud Computing, Socia...
2018
-
[27]
Juliet Test Suite for C/C++,
A. Richardson, “Juliet Test Suite for C/C++, ” https://github.com/arich ardson/juliet-test-suite-c
-
[28]
Objdump,
I. Free Software Foundation, “Objdump, ” https://man7.org/linux/man- pages/man1/objdump.1.html
-
[29]
Beyond the c: Retargetable decompila- tion using neural machine translation,
I. Hosseini and B. Dolan-Gavitt, “Beyond the c: Retargetable decompila- tion using neural machine translation, ”arXiv preprint arXiv:2212.08950, 2022
2022 arXiv
-
[30]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need, ” inAdvances in neural information processing systems , 2017
2017
-
[31]
Codebert: A pre-trained model for programming and natural languages,
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages, ” 2020
2020
-
[32]
Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,
Y. Wang, W. Wang, S. Joty, and S. C. H. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, ” 2021
2021
-
[33]
Codet5+: Open code large language models for code understanding and generation,
Y. Wang, H. Le, A. D. Gotmare, N. D. Q. Bui, J. Li, and S. C. H. Hoi, “Codet5+: Open code large language models for code understanding and generation, ” 2023
2023
-
[34]
Fairseq,
Meta AI, “Fairseq, ” https://ai.facebook.com/tools/fairseq/
-
[35]
Codegpt: Openai gpt for code generation,
H. Face, “Codegpt: Openai gpt for code generation, ” 2020, https://hugg ingface.co/microsoft/CodeGPT-small-py
2020
-
[36]
Using recurrent neural networks for decompilation,
D. S. Katz, J. Ruchti, and E. Schulte, “Using recurrent neural networks for decompilation, ” in2018 IEEE 25th International Conference on Soft- ware Analysis, Evolution and Reengineering (SANER) . IEEE, 2018
2018
-
[37]
Roberta: A robustly optimized bert pretraining approach,
Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized bert pretraining approach, ” 2019
2019
-
[38]
Does bleu score work for code migration?
N. Tran, H. Tran, S. Nguyen, H. Nguyen, and T. Nguyen, “Does bleu score work for code migration?” in 2019 IEEE/ACM 27th International Conference on Program Comprehension (ICPC) . IEEE, 2019
2019
-
[39]
Who evaluates the evaluators? on automatic metrics for assessing ai-based offensive code generators,
P. Liguori, C. Improta, R. Natella, B. Cukic, and D. Cotroneo, “Who evaluates the evaluators? on automatic metrics for assessing ai-based offensive code generators, ”Expert Systems with Applications, 2023
2023
-
[40]
Automating code-related tasks through transformers: The impact of pre-training,
R. Tufano, L. Pascarella, and G. Bavota, “Automating code-related tasks through transformers: The impact of pre-training, ” 2023
2023
-
[41]
Flawfinder: a static analysis tool for finding vulnera- bilities in c/c++ source code,
D. A. Wheeler, “Flawfinder: a static analysis tool for finding vulnera- bilities in c/c++ source code, ” https://dwheeler.com/flawfinder/
-
[42]
A unified approach to interpreting model predictions,
S. M. Lundberg and S.-I. Lee, “A unified approach to interpreting model predictions, ” in Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, Eds. Curran Associates, Inc.,
-
[43]
The clever hans effect in anomaly detection,
J. Kauffmann, L. Ruff, G. Montavon, and K.-R. Müller, “The clever hans effect in anomaly detection, ” 2020
2020
-
[44]
Flawfinder: Cwe-compatible,
D. A. Wheeler, “Flawfinder: Cwe-compatible, ” https://dwheeler.com/f lawfinder/flawfinder.pdf
-
[45]
T. L. Foundation, “LLVM, ” https://llvm.org/
-
[46]
CVE-2014-0160 Detail,
NIST, “CVE-2014-0160 Detail, ” https://nvd.nist.gov/vuln/detail/cve- 2014-0160
2014
-
[47]
The Heartbleed Bug,
I. Synopsis, “The Heartbleed Bug, ” https://heartbleed.com/0
-
[2017]
[Online]. Available: http://papers.nips.cc/paper/7062-a-unified- approach-to-interpreting-model-predictions.pdf EuroSec ’25, March 30–April 3, 2025, Rotterdam, Netherlands Domenico Cotroneo, Francesco C. Grasso, Roberto Natella, Vittorio Orbinato
2025
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.