REVIEW 3 major objections 7 minor 33 references
The Strengths and Behavioral Quirks of Java Bytecode Decompilers
T0 review · 3 major / 7 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read The paper measures eight Java decompilers on 2,041 real-world classes and finds that no single tool produces correct, behavior-preserving source code for all of them, while a naive ensemble of decompilers outperforms every individual tool.
desk verdict A solid, honest empirical study of eight Java decompilers on 2041 real-world classes; the headline semantic rates are upper bounds because they mean 'passes the existing tests,' but the no-single-tool conclusion holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is equivalence modulo inputs (EMI), borrowed from compiler validation: a decompiled program is judged semantically equivalent to the original if it passes the original project's test suite. The pipeline compiles source to bytecode with javac or ecj, decompiles each class, recompiles the result, compares original and recompiled bytecode with Jardiff for strict equivalence, and runs the project's tests on the recompiled classes; syntactic distortion is measured separately as the minimum number of edits to turn the original AST into the decompiled AST. EMI converts semantic evaluation from a manual task into an automated, scalable check, which is what makes a study at this scale possible.
What would settle it
Re-run the experiment's semantic-equivalence check on the classes that a decompiler passes, but replace the original test suite with coverage-guided or randomly generated inputs that exercise the differing bytecode paths; if a measurable fraction of the 78% set then fails, the reported semantic-equivalence rates are optimistic.
Extended reading notes
Core claim
The paper's central claim is that Java decompilation is a diverse, imperfect art: because compilation irreversibly discards information, each decompiler reconstructs the missing structure through different assumptions, and no single implementation covers the full space of real-world bytecode. Using an automated equivalence-modulo-inputs pipeline, the paper shows that the highest-scoring decompiler, Procyon, produces semantically equivalent code for 78% of test-covered classes, while a naive multi-decompiler ensemble reaches 91% because different tools succeed on disjoint sets of classes, with 276 classes handled correctly by only one of six decompilers and 157 classes by none. The paper further classifies deceptive decompilation as a distinct failure mode and reports that even the best tools exhibit it.
Load-bearing premise
The entire semantic-equivalence measurement assumes that passing the project's existing test suite demonstrates that decompiled code behaves like the original; when tests do not cover the constructs a decompiler changed, behavior-changing output is silently counted as equivalent.
Editorial extensions
If this is right
- Decompiler users cannot safely treat any single tool's output as authoritative: even the best tool produces behavior-changing output on some fraction of classes.
- Trying multiple decompilers in a quality-ranked order is a cheap way to gain 11 points of recompilability and 13 points of test-passing coverage over the best single decompiler.
- Decompiler maintainers can embed the EMI-based pipeline in their regression testing; the study already led to two fixed bugs and three acknowledged ones.
- The 157 classes that no tested decompiler handles correctly constitute a concrete, reusable benchmark for future decompilation research.
- Tools that minimize syntactic distortion are not the same as tools that maximize recompilability, so the right decompiler depends on whether the goal is human reading or further compilation.
Reading between the lines
- Because the test-suite proxy is only as strong as its coverage, the true rate of behavior-preserving decompilation is probably lower than 78%, and the paper's contribution is the measurement mechanism rather than a precise upper bound on decompiler quality.
- The Multi-DC result suggests an obvious product: an ensemble decompiler that selects or merges per-method outputs could plausibly push coverage beyond 95%, since failures are concentrated in 157 classes rather than spread uniformly.
- The same EMI pipeline could be transferred to other bytecode formats such as Android DEX or .NET CIL, or to obfuscated Java bytecode, where the diversity of decompiler assumptions is likely to matter even more.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a fully automated empirical evaluation of eight Java bytecode decompilers (CFR, Dava, Fernflower, JADX, JD-Core, Jode, Krakatau, Procyon) on 14 open-source projects totaling 2,041 Java classes. The pipeline compiles each project with javac and ecj, decompiles the resulting class files, and assesses output along four dimensions: syntactic correctness (recompilability), syntactic distortion (AST edit distance from the original), strict bytecode equivalence, and semantic equivalence modulo inputs (passing the original project's test suite on test-covered classes). The main findings are that no single decompiler handles all classes: Procyon achieves the best single-tool results with about 84% syntactically correct and 78% semantically equivalent outputs, while a naive ensemble (Multi-DC) raises those figures to about 95% and 91%. The authors also report deceptive decompilation cases and manually triaged several decompiler bugs.
Significance. If the results hold, this is a useful and substantial empirical contribution: it is an order of magnitude larger than prior decompiler comparisons, it adapts equivalence-modulo-inputs testing to decompilation, and it provides a publicly available, reproducible pipeline and dataset. The central qualitative claim that no single modern decompiler can correctly handle the variety of real-world Java bytecode is well supported by the recompilation and test-passing measurements. The paper's main quantitative contribution, however, is the semantic-equivalence rates, and those are directly bounded by the quality and coverage of the original test suites. The authors acknowledge this threat but do not quantify it, so the exact figures should be treated as upper bounds rather than precise measurements of behavioral equivalence. The syntactic-correctness results and the Multi-DC comparison are more robust and likely to survive even a stricter semantic evaluation.
major comments (3)
- [§III-A, Definition 3; §IV-B, Figure 3; §V-a] The semantic-equivalence metric is defined as passing the original test suite, and the RQ2 evaluation is restricted to the 2,397 class files covered by at least one test. The paper does not quantify per-class test adequacy: a class counts as covered if a test touches it at all, so a decompiler may alter untested methods and still be labeled equivalent. Consequently, the headline rates (Procyon 78%, Multi-DC 91%; Figure 3 and Table III) are upper bounds under the current test suites, and the amount by which they would change under stronger coverage is unknown. The internal-validity paragraph acknowledges that test-suite quality influences the observations, but it does not bound the effect. Please report coverage statistics (e.g., method/line coverage per class and the distribution of the number of tests per class) and, if possible, recompute the rates on classes meeting a minimum coverage threshold.
- [§IV-A, Figure 2, Table III] The summary table mixes denominators: the #Recompilable ratios are computed over all class-compiler pairs (2,041 javac pairs plus 1,887 ecj pairs, i.e., 3,928), while the #PassTest ratios are computed only over the 2,397 test-covered pairs. The table gives no denominator row or note, so a reader cannot tell that the 84% and 78% figures refer to different populations. In addition, the Procyon counts in the text (1,609/1,887 = 85.3% for javac; 1,532/1,887 = 81.2% for ecj) do not match Figure 2 (85.7% and 81.2%, respectively). Please state the denominators explicitly in Table III and reconcile the RQ1 numbers between the text and the figure.
- [§III-A, Definition 2; §IV-D, Figure 5] Definition 2 defines syntactic distortion as the minimum number of atomic edits to transform the AST of the original into the AST of the decompiled version, which includes renaming edits; however, Section IV-D and Figure 5 state that renaming is excluded from the measurement. Since variable renaming is one of the most visible differences between decompilers, the reported distortion values understate the dissimilarity and do not correspond to the definition as written. The definition or the experimental procedure should be made consistent, and the metric description should clearly state whether identifier changes contribute to the edit distance.
minor comments (7)
- [§VII, Conclusion] The conclusion says the projects were 'compiled with 2 different decompilers'; this should say 'compilers'.
- [§III-D, Study Subjects] The statement 'This represents 1887 class files for each compiler' is ambiguous because commons-lang was compiled only with javac; please state the per-compiler denominators explicitly (2,041 for javac, 1,887 for ecj).
- [§IV-A, RQ1] The text reports p-values for the chi-square test but mentions a 99% confidence level; it would be clearer to state the significance level and interpret the p-values directly against it.
- [Figure 3] The bars show counts but not percentages; adding percentage labels or a second axis would make the ranking by semantic-equivalence rate easier to read, especially since the denominator (2,397) is fixed.
- [§IV-E, RQ5] The phrase 'We exclude Dava and Krakatau because they that do not handle correctly any unique class file' contains a grammatical error and should be rephrased as 'because they do not handle any unique class file correctly.'
- [Table I] The #LOC column includes Python files for Krakatau, but the column heading says 'line of code in all Java files'; clarify that the count is language-inclusive or rename the column.
- [References] Reference [17] has a malformed URL ('https://http://java-decompiler.github.io/'); it should be 'https://java-decompiler.github.io/'.
Circularity Check
Secondary Multi-DC superiority is partly forced by in-sample ordering; the main no-single-decompiler finding is independent.
-
fitted input called prediction
[Section IV-E (RQ5) and Table III]
"we have implemented a naive Multi-DC that uses each decompiler one by one until it finds a syntactically correct decompilation result. The order of decompiler tried follows a ranking by decreasing success rate according to the six most successful decompilers in terms of semantic equivalence modulo inputs rates. ... Overall, the naive Multi-DC implementation performs the best in terms of both syntactically correct and semantically equivalent modulo inputs criteria."
The ranking used to order Multi-DC is computed on the same 2,397 test-covered classes on which Multi-DC is then evaluated, and Multi-DC stops at the first decompiler that yields recompilable output. Consequently its set of passing classes necessarily contains the passing set of the first-ranked decompiler, and its recompilable set contains that decompiler's recompilable set. Since the first-ranked decompiler is chosen precisely because it has the highest observed pass rate, Multi-DC's 'outperforms every single decompiler' conclusion is a monotonicity consequence of the construction rather than an independent empirical discovery. The 11- and 13-point gains are in-sample and partly forced; an out-of-sample or ablation comparison would be needed to measure a genuine diversity benefit.
full rationale
The paper's primary claim—that no single decompiler correctly handles the variety of real-world bytecode, with the best recompiling 84% and passing original tests for 78% of classes—is an external empirical measurement. The decompilers, compilers, projects, and test suites are independent of the authors' conclusions, and Definition 3 is an explicit operationalization of semantic equivalence modulo inputs as passing the original test suite. That operationalization creates a test-coverage validity threat (acknowledged in Section V-a), but it is not a circular derivation: the measured pass rates are not assumed in order to define the metric. The AST-diff, bytecode-diff, and coverage tooling, including some from the authors' prior work, are publicly available implementation tools and are not used to assert the conclusion. The one genuine construction issue is confined to RQ5: the Multi-DC ordering is fitted to the same data on which Multi-DC is evaluated, and the ensemble's superiority is guaranteed by set containment once the best-performing decompiler is placed first. This makes the diversity-boost result partly circular, but it does not undermine the independent finding that no single decompiler is fully effective. Overall circularity is therefore moderate and localized to a secondary result.
Assumptions & free parameters
assumptions (6)
- domain assumption A class that passes the original project's test suite is semantically equivalent modulo inputs to the original (Definition 3).
- domain assumption Recompilability with javac or ecj is a valid operational definition of syntactic correctness (Definition 1).
- domain assumption AST edit distance captures syntactic distortion and human readability (Definition 2).
- domain assumption The 14 selected projects are representative of real-world Java programs.
- domain assumption The 20-minute test timeout is sufficient to classify non-termination without distorting results.
- domain assumption Manually excluding tests that already fail on the original project does not bias the equivalence measurement.
Cite this review
Pith. "Pith review of The Strengths and Behavioral Quirks of Java Bytecode Decompilers." pith.science (2026). https://pith.science/paper/6SEDHSZ7
@misc{pith2026190806895,
author = {Pith},
title = {Pith review of: The Strengths and Behavioral Quirks of Java Bytecode Decompilers},
year = {2026},
howpublished = {\url{https://pith.science/paper/6SEDHSZ7}},
note = {Machine review of arXiv:1908.06895}
}
read the original abstract
During compilation from Java source code to bytecode, some information is irreversibly lost. In other words, compilation and decompilation of Java code is not symmetric. Consequently, the decompilation process, which aims at producing source code from bytecode, must establish some strategies to reconstruct the information that has been lost. Modern Java decompilers tend to use distinct strategies to achieve proper decompilation. In this work, we hypothesize that the diverse ways in which bytecode can be decompiled has a direct impact on the quality of the source code produced by decompilers. We study the effectiveness of eight Java decompilers with respect to three quality indicators: syntactic correctness, syntactic distortion and semantic equivalence modulo inputs. This study relies on a benchmark set of 14 real-world open-source software projects to be decompiled (2041 classes in total). Our results show that no single modern decompiler is able to correctly handle the variety of bytecode structures coming from real-world programs. Even the highest ranking decompiler in this study produces syntactically correct output for 84% of classes of our dataset and semantically equivalent code output for 78% of classes.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
T. Lindholm, F. Yellin, G. Bracha, and A. Buckley, The Java Virtual Machine Specification. Pearson Education, 2014
work page 2014
-
[2]
Meaningful Variable Names for Decompiled Code: A Machine Transla- tion Approach,
A. Jaffe, J. Lacomis, E. J. Schwartz, C. L. Goues, and B. Vasilescu, “Meaningful Variable Names for Decompiled Code: A Machine Transla- tion Approach,” in26th Conference on Program Comprehension (ICPC), (New York, NY , USA), pp. 20–30, ACM, 2018
work page 2018
-
[3]
G. Nolan, Decompiler Design , pp. 121–157. Berkeley, CA: Apress, 2004
work page 2004
-
[4]
Using Compilation/Decompilation to Enhance Clone Detection,
C. Ragkhitwetsagul and J. Krinke, “Using Compilation/Decompilation to Enhance Clone Detection,” in 11th International Workshop on Software Clones (IWSC), pp. 1–7, Feb 2017
work page 2017
-
[5]
Helping Johnny to Analyze Malware: A Usability-Optimized Decompiler and Malware Analysis User Study,
K. Yakdan, S. Dechand, E. Gerhards-Padilla, and M. Smith, “Helping Johnny to Analyze Malware: A Usability-Optimized Decompiler and Malware Analysis User Study,” in IEEE Symposium on Security and Privacy (SP), pp. 158–177, May 2016
work page 2016
-
[6]
PsybOt Malware: A Step- By-Step Decompilation Case Study,
L. ˇDurfina, J. K ˇroustek, and P. Zemek, “PsybOt Malware: A Step- By-Step Decompilation Case Study,” in 20th Working Conference on Reverse Engineering (WCRE) , pp. 449–456, Oct 2013
work page 2013
-
[7]
An Empirical Approach to Software Archaeology,
G. Robles, J. M. Gonzalez-Barahona, and I. Herraiz, “An Empirical Approach to Software Archaeology,” in 21st International Conference on Software Maintenance (ICSM) , pp. 47–50, 2005
work page 2005
-
[8]
An Evaluation of Current Java Bytecode Decompilers,
J. Hamilton and S. Danicic, “An Evaluation of Current Java Bytecode Decompilers,” in 9th IEEE International Working Conference on Source Code Analysis and Manipulation (SCAM) , pp. 129–136, Sep. 2009
work page 2009
Show all 33 references
-
[9]
An Evaluation of Output from Current Java Bytecode Decompilers: Is it Android Which is Responsible for Such Quality Boost?,
J. Kostelanský and L. Dedera, “An Evaluation of Output from Current Java Bytecode Decompilers: Is it Android Which is Responsible for Such Quality Boost?,” in Communication and Information Technologies (KIT), pp. 1–6, Oct 2017
2017
-
[10]
Compiler Validation via Equivalence Modulo Inputs,
V . Le, M. Afshari, and Z. Su, “Compiler Validation via Equivalence Modulo Inputs,” in 35th Conference on Programming Language Design and Implementation (PLDI), (New York, NY , USA), pp. 216–226, ACM, 2014
2014
-
[11]
Fine-grained and Accurate Source Code Differencing,
J.-R. Falleri, F. Morandat, X. Blanc, M. Martinez, and M. Monperrus, “Fine-grained and Accurate Source Code Differencing,” in 29th Inter- national Conference on Automated Software Engineering (ASE) , (New York, NY , USA), pp. 313–324, ACM, 2014
2014
-
[12]
Hunting for Bugs in Code Coverage Tools via Randomized Differential Testing,
Y . Yang, Y . Zhou, H. Sun, Z. Su, Z. Zuo, L. Xu, and B. Xu, “Hunting for Bugs in Code Coverage Tools via Randomized Differential Testing,” in 41st International Conference on Software Engineering (ICSE) , ACM, 2019
2019
-
[13]
Benfield, “CFR.” https://www.benf.org/other/cfr/, 2019
L. Benfield, “CFR.” https://www.benf.org/other/cfr/, 2019. [Online; accessed 19-July-2019]
2019
-
[14]
L. J. H. Jerome Miecznikowski, Nomair A. Naeem, “Dava.” http://www. sable.mcgill.ca/dava/, 2019. [Online; accessed 19-July-2019]
2019
-
[15]
Fernflower
“Fernflower.” https://github.com/JetBrains/intellij-community/tree/ master/plugins/java-decompiler/engine, 2019. [Online; accessed 19-July-2019]
2019
-
[16]
[Online; accessed 19-July-2019]
skylot, “JADX.” https://github.com/skylot/jadx, 2019. [Online; accessed 19-July-2019]
2019
-
[17]
Java Decompiler
E. Dupuy, “Java Decompiler.” https://http://java-decompiler.github.io/,
-
[18]
Hoenicke, “JODE.” http://jode.sourceforge.net/, 2019
J. Hoenicke, “JODE.” http://jode.sourceforge.net/, 2019. [Online; ac- cessed 19-July-2019]
2019
-
[19]
Krakatau
Storyyeller, “Krakatau.” https://github.com/Storyyeller/Krakatau, 2019. [Online; accessed 19-July-2019]
2019
-
[20]
Procyon
M. Strobel, “Procyon.” https://bitbucket.org/mstrobel/procyon, 2019. [Online; accessed 19-July-2019]
2019
-
[21]
Spoon: A Library for Implementing Analyses and Transformations of Java Source Code,
R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “Spoon: A Library for Implementing Analyses and Transformations of Java Source Code,” Software: Practice and Experience , vol. 46, pp. 1155–1179, 2015
2015
-
[22]
Decompiling Java Bytecode: Prob- lems, Traps and Pitfalls,
J. Miecznikowski and L. Hendren, “Decompiling Java Bytecode: Prob- lems, Traps and Pitfalls,” in Compiler Construction (R. N. Horspool, ed.), (Berlin, Heidelberg), pp. 111–127, Springer Berlin Heidelberg, 2002
2002
-
[23]
Metrics for Measuring the Effectiveness of Decompilers and Obfuscators,
N. A. Naeem, M. Batchelder, and L. Hendren, “Metrics for Measuring the Effectiveness of Decompilers and Obfuscators,” in 15th IEEE International Conference on Program Comprehension (ICPC), pp. 253– 258, June 2007
2007
-
[24]
An Analysis on Java Programming Language Decompiler Capabilities,
K. Gusarovs, “An Analysis on Java Programming Language Decompiler Capabilities,” Applied Computer Systems , vol. 23, no. 2, pp. 109–117, 2018
2018
-
[25]
An Experimentation Framework for Evaluating Disassembly and De- compilation Tools for C++ and Java,
L. Vinciguerra, L. Wills, N. Kejriwal, P. Martino, and R. Vinciguerra, “An Experimentation Framework for Evaluating Disassembly and De- compilation Tools for C++ and Java,” in 10th Working Conference on Reverse Engineering (WCRE) , (Washington, DC, USA), pp. 14–, IEEE Computer...
2003
-
[26]
Speculative disassembly of binary code,
M. A. B. Khadra, D. Stoffel, and W. Kunz, “Speculative disassembly of binary code,” in International Conference on Compliers, Architectures, and Sythesis of Embedded Systems (CASES) , pp. 1–10, Oct 2016
2016
-
[27]
Towards Neural Decompilation,
O. Katz, Y . Olshaker, Y . Goldberg, and E. Yahav, “Towards Neural Decompilation,” arXiv e-prints, p. arXiv:1905.08325, May 2019
1905 arXiv
-
[28]
Reconstruction of Com- posite Types for Decompilation,
K. Troshina, Y . Derevenets, and A. Chernov, “Reconstruction of Com- posite Types for Decompilation,” in 10th IEEE Working Conference on Source Code Analysis and Manipulation (SCAM) , pp. 179–188, Sep. 2010
2010
-
[29]
M. J. Van Emmerik, Static Single Assignment for Decompilation . Uni- versity of Queensland, 2007
2007
-
[30]
Using Recurrent Neural Networks for Decompilation,
D. S. Katz, J. Ruchti, and E. Schulte, “Using Recurrent Neural Networks for Decompilation,” in 25th International Conference on Software Anal- ysis, Evolution and Reengineering (SANER) , pp. 346–356, March 2018
2018
-
[31]
Evolv- ing exact decompilation,
E. Schulte, J. Ruchti, M. Noonan, D. Ciarletta, and A. Loginov, “Evolv- ing exact decompilation,” in Workshop on Binary Analysis Research (Y . Shoshitaishvili and R. F. Wang, eds.), (San Diego, CA, USA), Feb. 18-21 2018
2018
-
[32]
EnFuzz: Ensemble Fuzzing with Seed Synchronization among Diverse Fuzzers,
Y . Chen, Y . Jiang, F. Ma, J. Liang, M. Wang, C. Zhou, Z. Su, and X. Jiao, “EnFuzz: Ensemble Fuzzing with Seed Synchronization among Diverse Fuzzers,” arXiv e-prints, p. arXiv:1807.00182, Jun 2018
2018 arXiv
-
[2019]
[Online; accessed 19-July-2019]
2019
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.