REVIEW 4 major objections 4 minor 1 cited by
IDOL: Improved Different Optimization Levels Testing for Solidity Compilers
T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Rewriting Solidity contracts back into unoptimized form before differential compilation exposes three real compiler optimization bugs.
desk verdict A plausible Solidity-specific extension of DOL testing with three reported compiler bugs, but the preprint gives too little detail to verify the central claim; worth sending to review only if the authors can supply artifacts and precise transformation rules. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The engine of IDOL is the reverse optimization transformation: a rewrite that takes a code shape that looks like the output of an optimization and turns it back into the unoptimized input shape, for example moving a loop-invariant computation back inside a loop, or converting an inverted loop back into a plain while loop. These rewrites are claimed to preserve the contract's semantics exactly, while presenting the compiler with code that is not already in optimized form. When the compiler is then run with different --optimize settings, the variant forces it to perform the corresponding optimization from scratch, maximizing the chance that a defect in that optimization path will produce an observable output difference.
What would settle it
Take any one of the three reported bugs, compile the same variant with optimization disabled, and compare its execution result with the original untransformed contract also compiled with optimization disabled; any difference in behavior between variant and original would show that a transformation changed semantics, making the reported bug a false positive rather than a compiler defect.
Extended reading notes
Core claim
The paper's central discovery is stated in Section 3: over 160,000 test programs generated from Solidity contracts, IDOL identified three confirmed compiler optimization bugs, whereas the original DOL approach detected none of these three. The three bugs are the optimizer Keccak cache bug, the traditional code generation pipeline bug, and the FullInliner non-expression split parameter evaluation order bug. Each shows up only when a semantically equivalent variant, created by reversing a known optimization (such as loop-invariant code motion or loop inversion), is compiled and executed at different optimization settings. The author's conclusion is that deliberately feeding the optimizer its own "input side" expands the set of triggerable optimization paths.
Load-bearing premise
The method assumes that the reverse optimization transformations never change the behavior of a contract, including edge cases like overflow, gas, and storage, so any output difference must come from the compiler.
Editorial extensions
If this is right
- IDOL can serve as an oracle-free testing pipeline for Solidity compiler releases, since it requires no hand-written expected outputs.
- The three confirmed bugs point to concrete defect-prone optimizer components: Keccak caching, the traditional code-generation pipeline, and FullInliner expression splitting.
- Compiler developers can add IDOL-generated variants to regression suites to prevent these classes of miscompilation from reappearing.
- The results suggest that reverse optimization transformations increase the diversity of optimization-triggering inputs compared with raw generated programs.
Reading between the lines
- Editorial inference: the same reverse-optimization recipe should transfer to other smart-contract compilers, such as Vyper, or to general-purpose compilers if the transformation rules are instantiated per target.
- Editorial inference: because the paper does not prove semantic equivalence, a fair re-check should run each variant against its original under identical settings; until then the three bug reports carry a hidden assumption.
- Editorial inference: the number of bugs found could be used as a rough sensitivity measure, and expanding the rule set with more Solidity-specific reverse transformations should raise the hit rate; this is testable by rerunning IDOL with an expanded rule set.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. IDOL is a differential testing method for the Solidity compiler. Starting from contracts generated by Solsmith, it applies transformations that are the reverse of known compiler optimizations (e.g., moving loop-invariant code back into the loop and converting a do-while back into a while loop) to create semantically equivalent variants, then compiles and runs each variant under three solc configurations (--optimize=false, --optimize-runs=1, and --optimize-runs=200) and compares execution outputs. The paper reports a preliminary evaluation of 160,000 Solsmith-generated programs in which IDOL found three compiler optimization bugs, named Keccak cache, traditional code-generation pipeline, and FullInliner parameter evaluation order, and states that the original DOL approach did not detect these bugs. The contribution is a heuristic for increasing the chance that differential testing triggers compiler optimization logic, with the central claim being that this heuristic finds real Solidity compiler bugs that plain DOL misses.
Significance. If the three reported bugs are genuine and reproducible, the result is a useful and credible data point: IDOL is a lightweight extension of DOL, built on top of an existing contract generator, and it found optimization bugs that plain DOL missed on the same corpus. The method has no fitted parameters, and the differential oracle is external (comparing solc behavior at different optimization levels), so the core design is not circular. However, the evidence in the manuscript is at the level of a preliminary report: none of the three bug descriptions includes enough information to be independently reproduced, and there is no false-positive analysis, no trigger-rate measurement, and no validation that the reverse transformations preserve semantics on the executed inputs. The contribution is therefore plausible but not yet established at the standard of a full research paper.
major comments (4)
- [Section 2, Figures 1 and 2] The semantic-equivalence assumption is the load-bearing premise of IDOL, but the paper provides only informal examples and no side conditions. Loop-invariant code motion is valid only if the moved computation (e.g., x = y + z and x * x in Figure 1) has no operands modified in the loop and no side effects; loop inversion as shown in Figure 2 is not semantics-preserving for arbitrary while loops containing break, continue, or conditions with side effects. The manuscript never explains how the transformations are guarded, nor does it report any validation (e.g., differential execution of the original and variant at the same optimization level) that the generated variants are equivalent on the executed inputs. Without such validation, an output discrepancy could be caused by the transformation itself rather than by a compiler bug, which would make the three reported bugs false positives.
- [Section 3, bug descriptions] The three central bug reports are single-sentence descriptions with no solc version, no minimized contract, no issue-tracker identifier, no diff of the affected bytecode, and no executable reproducer. Since the claim that these are 'confirmed' bugs is the paper's main evidence of effectiveness, these details must be supplied (or provided in a supplementary artifact) before the claim can be independently checked. The phrase 'at the time of writing' in the abstract and in Section 3 is also not verifiable; the authors should name a specific solc release and provide linkable bug reports.
- [Section 3, final paragraph] The comparison with the original DOL approach is under-specified. The paper reports only that DOL 'does not detect' the three bugs on the same 160,000 programs; it does not state how many test executions or output comparisons were performed, how many discrepancy candidates DOL found, whether the same compilation and execution oracle was used for both methods, or how many IDOL variants were generated per original program. Reporting trigger rates and the total number of discrepancy candidates for both methods would let the reader judge whether the improvement is real or an artifact of small numbers.
- [Section 3, Figure 3] The testing process is described only at the level of a flow diagram: 'Compilation and Deployment' and 'Test Results' are mentioned, but the paper never defines what is executed (which functions, with what arguments and transaction sequence) or how output results are compared. This missing oracle definition makes the experiment impossible to reproduce from the manuscript text alone, and it also leaves unspecified how discrepancies are triaged. The authors should state the exact comparison procedure and the criterion for classifying a discrepancy as a confirmed compiler bug.
minor comments (4)
- [Section 3, first paragraph] The text 'sloc compiles them' appears to be a typo; it should read 'solc compiles them'.
- [Figure 3] The labels 'Optimization Level L1' and 'Optimization Level L2' are undefined; the paper should map these to the three concrete configurations (--optimize=false, --optimize-runs=1, --optimize-runs=200) described in Section 2.
- [References] Reference [3] is missing a publication venue and year, and reference [4] appears to be a preprint or accepted paper with no bibliographic details; both should be completed for reproducibility of the related-work discussion.
- [Section 3, bug names] The name 'Traditional Code Generation Pipeline Bug' is vague and does not identify a specific solc component or optimization pass; consider using a more precise description or the actual issue title if the bug is tracked.
Circularity Check
No circularity: IDOL's central claim is an empirical differential-testing result, not a derivation from its own inputs.
full rationale
The paper's central claim is that IDOL, by applying reverse optimization transformations before different-optimization-level testing, detected three Solidity compiler optimization bugs that the original DOL approach did not detect. This is an empirical claim about externally observed compiler behavior, not a formal derivation that reduces to its assumptions. There are no fitted parameters being renamed as predictions: the three bug reports are outputs of a differential testing procedure comparing execution results at different optimization levels. The semantic-equivalence assumption of the reverse transformations is an unvalidated premise, and the lack of minimized reproducers and bug IDs makes the results hard to verify independently, but this is a correctness/validity threat, not circularity. The only self-citation is reference [4], used to support the general statement that many compiler errors are triggered during the optimization phase; this claim is not the paper's specific contribution, and the citation is not load-bearing. The finding of three bugs, if accurate, is an independent empirical outcome that could in principle be false, so the derivation chain is not circular. The comparison against the original DOL approach is a direct experimental comparison and does not presuppose the superiority of IDOL. Therefore, no circular step can be exhibited from the paper's text, and the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Semantically equivalent programs compiled at different optimization levels must produce identical execution results.
- domain assumption The reverse transformations in IDOL preserve the observable semantics of the smart contracts.
- domain assumption Solsmith-generated contracts are valid, representative Solidity programs.
Cite this review
Pith. "Pith review of IDOL: Improved Different Optimization Levels Testing for Solidity Compilers." pith.science (2026). https://pith.science/paper/HXWEZXTN
@misc{pith2026250612760,
author = {Pith},
title = {Pith review of: IDOL: Improved Different Optimization Levels Testing for Solidity Compilers},
year = {2026},
howpublished = {\url{https://pith.science/paper/HXWEZXTN}},
note = {Machine review of arXiv:2506.12760}
}
read the original abstract
As blockchain technology continues to evolve and mature, smart contracts have become a key driving force behind the digitization and automation of transactions. Smart contracts greatly simplify and refine the traditional business transaction processes, and thus have had a profound impact on various industries such as finance and supply chain management. However, because smart contracts cannot be modified once deployed, any vulnerabilities or design flaws within the contract cannot be easily fixed, potentially leading to significant financial losses or even legal issues. The compiler, as a critical component in the development process, directly affects the quality and security of smart contracts. This paper innovatively proposes a method, known as the Improved Different Optimization Levels (IDOL), for testing the Solidity compiler. The key idea behind IDOL is to perform reverse optimization transformations (i.e., change optimized form into unoptimized form) to generate semantically equivalent variants of the smart contracts under test, aiming to maximize the opportunities to trigger the optimization logic of compilers. We conducted a preliminary evaluation of IDOL and three confirmed compiler optimization bugs have been uncovered at the time of writing.
Figures
Forward citations
Cited by 1 Pith paper
-
Clinically-guided Data Synthesis for Laryngeal Lesion Detection
The abstract reports that 10% synthetic data improves laryngeal lesion detection by 9% internally and 22.1% externally, but the submitted full text is a different paper.
Reference graph
Works this paper leans on
-
[4]
A. Groce et al. , ``Randomized differential testing as a prelude to formal verification,'' in ICSE , pp. 621--631
-
[1]
11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttex...
-
[2]
L. Li, Y. Liang, Z. Liu, and Z. Yu, ``Understanding solidity event logging practices in the wild,'' in ESEC/FSE , p. 300–312, 2023
work page 2023
-
[3]
E. T. Barr et al. , ``The oracle problem in software testing: A survey,'' TSE , vol. 41, no. 5, pp. 507--525, 2014
work page 2014
-
[5]
J. Wu, J. Zheng, Z. Yang, and Z. Yu, ``Compiler optimization testing based on optimization-guided equivalence transformations,'' in FSE , 2025
work page 2025
-
[6]
J. Chen et al. , ``An empirical comparison of compiler testing techniques,'' in ICSE , pp. 180--190, 2016
work page 2016
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.