Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Solsmith: Solidity Random Program Generator for Compiler Testing

T0 review · 4 major / 7 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Solsmith randomly generates valid Solidity programs that expose confirmed defects in the Solidity compiler, with four distinct bugs found in solc 0.8.0.

desk verdict A plausible new Solidity fuzzer with real bug reports, but the evaluation is thin and the consistency oracle may be unsound on evaluation order. read the letter →

arxiv 2506.03909 v1 pith:BFOMSCKU submitted 2025-06-04 cs.SE

Lantian Li , Zhihao Liu , Zhongxing Yu This is my paper
classification cs.SE
keywords SoliditycompilertestingrandomprogramgenerationdifferentialsmartcontractsdefectsundefinedbehavioroptimizerIR-basedcompilation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Solsmith, a random generator of Solidity test programs built to find defects in the Solidity compiler. The authors argue that earlier work on Solidity compiler testing focused on testing methodology and left the harder problem of program generation unaddressed. Solsmith is designed around three goals: consistency (each generated program has a single meaning), diversity (wide coverage of language features), and compliance (programs compile cleanly). The evaluation runs 100,000 generated programs under three compiler configurations and reports four distinct, confirmed defects in solc 0.8.0, demonstrating that the generator is effective.

What carries the argument

The central object is the Solsmith generator, whose mechanism is the combination of three design objectives with language-specific strategies. Consistency is achieved by avoiding the four documented Solidity undefined behaviors (inline-assembly length operations on storage arrays, function-parameter evaluation-order dependence, dangling references, and verbatim bytecode) and by suppressing behavioral differences between the old and via-IR code generators, such as repeated placeholder use and expression evaluation order. Diversity comes from a wide feature set and manually adjusted frequencies, including distorting frequencies so optimizations fire more often. Compliance is ensured by explicit type conversions, division-by-zero guards, array-bounds checks, and unchecked blocks for overflow-sensitive code. Differential testing across three solc configurations provides the oracle.

What would settle it

Re-running the four minimal reproducers on solc 0.8.0 with the stated flags and finding that a reported divergence is a documented, intended difference between the old and via-IR code generators, or that the divergence persists after the relevant compiler fix, would overturn the claim that all four are confirmed compiler defects.

Watch

Extended reading notes

Core claim

The central claim is that careful random generation of Solidity programs, guided by language-specific consistency and compliance strategies, can reveal genuine Solidity compiler defects. The paper reports four confirmed defects in solc 0.8.0: a FullInliner defect that reorders function-call arguments during inlining, a function-selector defect under the via-IR pipeline that drops side effects, a keccak256 optimizer defect that treats hashes of different lengths as equal, and an unchecked-block defect where the via-IR pipeline mishandles increment at uint256 maximum and decrement at minimum. For each defect the paper gives a minimal triggering program and a root-cause explanation.

Load-bearing premise

The evaluation assumes that Solsmith's consistency strategies fully eliminate undefined behavior and IR-dependent semantic differences, so any differential behavior across the three compiler configurations is a genuine compiler defect rather than a legitimate semantic difference.

Editorial extensions

If this is right

  • The four confirmed defects give the Solidity team minimal reproducers and root causes for optimizer and via-IR pipeline bugs.
  • Solidity compilers still harbor configuration-dependent bugs that only appear under certain optimizer step sequences or IR settings.
  • Compiler testing research for Solidity needs dedicated program generators, not just testing methods, because generic generators do not transfer to Solidity's semantics.
  • The generator's strategies can be extended to later compiler versions and additional optimization combinations to find further defects.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The frequency-distortion approach for triggering optimizer paths could be automated with search or learning to maximize bug-finding per generated program.
  • The reported defects suggest that differential testing across optimizer and runs settings, rather than just compiler versions, is a high-yield oracle for smart-contract compilers.
  • The same consistency strategies could be adapted to other smart-contract languages whose compilers offer both direct and IR-based code generation paths.
  • Because the compiler itself can emit keccak256 calls for mappings and indexed events, the keccak256 defect may affect real contracts, not just synthetic test cases.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper presents Solsmith, a random test program generator for the Solidity compiler. Its design is organized around three objectives—consistency, diversity, and compliance—and implements strategies that include avoiding undefined behaviors, mitigating IR-dependent semantic differences, increasing the frequency of optimizer-triggering constructs, and inserting runtime checks for type, division, and array-bound errors. The tool is evaluated by differential testing: 100,000 generated programs are run under three configurations of solc (standard, optimizer with runs=1, and optimizer with runs=2^32−1), and the paper reports four distinct defects in solc 0.8.0, with a minimized example and a root-cause discussion for each. The paper concludes that these results demonstrate the effectiveness and potential of Solsmith.

Significance. If the central claims are substantiated, Solsmith would be a valuable contribution to Solidity compiler testing, an area with relatively few dedicated program generators. The open-source release of the tool and the detailed description of generation strategies are concrete assets. However, the evaluation as presented does not yet establish the claimed effectiveness: the differential oracle is not convincingly sound, the link between Solsmith's generated programs and the four reported defects is not shown, and there is no baseline comparison to random generation or existing Solidity fuzzers. The paper therefore provides a useful design and an interesting set of candidate bugs, but the evidence for the headline contribution is incomplete.

major comments (4)
  1. [§4.2.2 and §5.2] The differential-testing oracle is not sound because the avoidance of expression-evaluation-order differences is incomplete. The strategy in §4.2.2 tracks only variables that have been incremented or decremented and replaces them with constants on reuse; it does not prevent side-effecting function calls from appearing as subexpressions of arithmetic or relational expressions, even though §4.1.3 explicitly includes function-call expressions that return results. Solidity leaves the evaluation order of expression children unspecified, and the legacy and viaIR code generators are known to differ on this point. Consequently, a cross-configuration output difference that involves such a call is not necessarily a compiler defect. The four reported defects must be re-examined under this criterion, or the generator's consistency strategy must be strengthened, before the claim that Solsmith reveals confirmed defects can be accepted.
  2. [§5.2.1–§5.2.4 and Figures 9–12] The manuscript does not establish that the four defect-triggering programs are outputs of Solsmith. Figure 9 is a standalone Yul program, not a Solidity contract; Figures 10–12 are Solidity snippets but no reduction process is described, and no statistics are given on how many Solsmith-generated programs triggered each defect before minimization. Without this link, the four 'confirmed defects' cannot be attributed to Solsmith's generation strategy.
  3. [§5] The evaluation lacks baseline comparisons and quantitative success metrics. To support the claim that Solsmith is effective, the paper should compare against a random generator that ignores the described strategies, or against an existing Solidity fuzzer such as FUZZOL, using the same differential-testing harness. It should also report the number of defect-triggering programs per 100,000, the false-positive rate, and runtime. Without such a baseline, the four defects cannot be attributed to the generator's design.
  4. [§5.2] The word 'confirmed' is not verifiable because the paper gives no upstream issue or pull-request identifiers from the Solidity repository. Providing these links is standard for compiler-testing papers and is necessary to substantiate that the reported defects are genuine and not false positives of the comparison setup.
minor comments (7)
  1. [§4.2.1] The text refers to 'John et al. [25]', but reference [25] is the YARPGen paper by Livinskii, Babokin, and Regehr; correct the citation.
  2. [§3.4] The heading 'Preventing Integer Overflow' is misleading because wrapping arithmetic inside an `unchecked` block is not prevention of overflow; it is a deliberate choice to use wrapping semantics. Rephrase to something like 'Avoiding overflow-induced reverts'.
  3. [§5.1] The three configurations are all variations of cross-optimization testing on the same compiler version, whereas the background section (§2.2) also describes cross-version and cross-compilation strategies, which are not used in the evaluation. Clarify why only cross-optimization is chosen.
  4. [§5.2.1] Figure 9 is written in Yul, not Solidity; state this in the caption and explain how a Solidity test program produced by Solsmith reduces to this Yul snippet.
  5. [§5.2.3] The term 'cse option' is not defined; state that this is the common-subexpression-elimination step of the Yul optimizer.
  6. [§5.2.1] The phrase 'non-expression-splitted' should be 'non-expression-split'.
  7. [§3.2 and §4.2.2] The text lists 'function argument evaluation order' as undefined behavior in §3.2, but §4.2.2 discusses it as an IR-dependent difference; make the relationship between these two concepts explicit.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: Solsmith is an empirical tool report whose claims are evaluated by differential testing against external solc configurations, with only a non-load-bearing self-citation.

full rationale

The paper's central claim is that Solsmith generates valid, diverse Solidity programs and that differential testing across compiler configurations revealed four defects. This claim is not derived from the paper's own definitions or from fitted parameters. The generation strategies in Sections 3 and 4 are design choices, not models fitted to the evaluation outcomes, and the reported defects in Section 5.2 are presented as independently observed behavioral differences with root-cause analyses. The only self-citation is reference [2], which supports background statements about smart-contract adoption in the introduction and is not load-bearing for the compiler-testing contribution. No uniqueness theorem, ansatz, or known result is imported from the authors' prior work to force a conclusion. The paper does not define any prediction in terms of its inputs, and the differential-testing oracle is an external behavior comparison rather than a circular self-check. Accordingly, no circular step is identified, and the score of 1 merely acknowledges the presence of one minor, non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on an empirical tool run rather than a mathematical derivation. The main assumptions are that differential testing is a valid oracle and that the generator successfully avoids the semantic ambiguities it targets. The listed free parameters are engineering settings, not values fitted to explain a scientific result.

free parameters (3)
  • Maximum allowed local variables (16) = 16
    Chosen to avoid EVM stack-too-deep errors; it is an engineering constraint rather than a fitted scientific parameter.
  • Maximum nesting depth = not specified
    Manually adjustable to balance expressiveness and compiler stack limits; exact value used in the evaluation is not reported.
  • Feature generation probabilities = not specified
    Manually adjusted to increase frequencies of certain constructs (e.g., keccak256, inline assembly); exact values are not reported.
assumptions (3)
  • domain assumption Differential testing across solc configurations is a valid oracle for compiler defects.
    Sections 2.2 and 5.1 assume that differences in output across the three configurations indicate a compiler defect rather than an intentional semantic difference.
  • domain assumption Generated programs avoid all undefined behaviors and IR-dependent semantic differences listed in Section 3.2.
    The evaluation relies on the generator's avoidance of undefined behavior; if incomplete, some observed discrepancies could be false positives.
  • domain assumption Solidity compiler versions 0.8.0 and later have the semantics described in the paper.
    The bug explanations, especially the Unchecked defect, assume that the via-IR and standard pipelines should behave identically for the generated programs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Solsmith: Solidity Random Program Generator for Compiler Testing." pith.science (2026). https://pith.science/paper/BFOMSCKU

@misc{pith2026250603909,
  author       = {Pith},
  title        = {Pith review of: Solsmith: Solidity Random Program Generator for Compiler Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BFOMSCKU}},
  note         = {Machine review of arXiv:2506.03909}
}
read the original abstract

Smart contracts are computer programs that run on blockchain platforms, with Solidity being the most widely used language for their development. As blockchain technology advances, smart contracts have become increasingly important across various fields. In order for smart contracts to operate correctly, the correctness of the compiler is particularly crucial. Although some research efforts have been devoted to testing Solidity compilers, they primarily focus on testing methods and do not address the core issue of generating test programs. To fill this gap, this paper designs and implements Solsmith, a test program generator specifically aimed at uncovering defects in Solidity compilers. It tests the compiler correctness by generating valid and diverse Solidity programs. We have designed a series of unique program generation strategies tailored to Solidity, including enabling optimizations more frequently, avoiding undefined behaviour, and mitigating behavioural differences caused by intermediate representations. To validate the effectiveness of Solsmith, we assess the effectiveness of the test programs generated by Solsmith using the approach of differential testing. The preliminary results show that Solsmith can generate the expected test programs and uncover four confirmed defects in Solidity compilers, demonstrating the effectiveness and potential of Solsmith.

Figures

Figures reproduced from arXiv: 2506.03909 by the authors.

Figure 1
Figure 1. Design Overview Second, the diversity of the test programs is enhanced by combining a wide range of language features, along with manually adjusting the frequency of these features. Finally, we design a series of strategies to ensure the compliance of the test programs, including avoiding type errors, preventing division by zero, preventing integer overflow, and preventing array out-of-bounds errors. On the other ha… view at source ↗
Figure 5
Figure 5. An Example of Preventing Integer Overflow. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. A Method to Avoid Stack-Too-Deep Errors. [PITH_FULL_IMAGE:figures/full_fig_p004_6.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: An Example of Preventing Array Index Out-of-Bounds [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 7
Figure 7. Figure 7: Choice of Function Parameters. later. With the release of Solidity major version 0.8, earlier versions of the compiler will gradually cease to be maintained. Consequently, certain syntax from versions prior to 0.8.0 may not be supported. At the same time, we aim to tes…
Figure 8
Figure 8. Figure 8: An Example of Function Parameter Evaluation Order. [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]
Figure 9
Figure 9. Figure 9: A Program that Triggers the FullInliner Defect. [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: A Program that Triggers the Selector Defect. [PITH_FULL_IMAGE:figures/full_fig_p009_10.png]
Figure 12
Figure 12. Figure 12: A Program that Triggers the Unchecked Defect. [PITH_FULL_IMAGE:figures/full_fig_p009_12.png]
Figure 11
Figure 11. Figure 11: A Program that Triggers the keccak256 Defect. [PITH_FULL_IMAGE:figures/full_fig_p009_11.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Clinically-guided Data Synthesis for Laryngeal Lesion Detection

    eess.IV 2025-08 unverdicted novelty 5.0 of 10

    The abstract reports that 10% synthetic data improves laryngeal lesion detection by 9% internally and 22.1% externally, but the submitted full text is a different paper.

Reference graph

Works this paper leans on

34 extracted references · 34 canonical work pages · cited by 1 Pith paper

  1. [1]

    Challenges and common solutions in smart contract development,

    N. Kannengiesser, S. Lins, K. Winter, and A. Sunyaev, “Challenges and common solutions in smart contract development,”TSE, vol. 48, no. 11, pp. 4291–4318, 2021

    work page 2021

  2. [2]

    Understanding solidity event logging practices in the wild,

    L. Li, Y . Liang, Z. Liu, and Z. Yu, “Understanding solidity event logging practices in the wild,” inPro- ceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oun- dations of Software Engineering, ESEC/FSE 2023, (New York, NY , USA), p. 300–312, Association for Computing Machinery, 2023

    work page 2023

  3. [3]

    A. M. Antonopoulos and G. Wood,Mastering ethereum: building smart contracts and dapps. 2018

    work page 2018

  4. [4]

    Ethereum: A secure decentralised gener- alised transaction ledger,

    G. Woodet al., “Ethereum: A secure decentralised gener- alised transaction ledger,”Ethereum project yellow paper, vol. 151, no. 2014, pp. 1–32, 2014

    work page 2014

  5. [5]

    Use of blockchain-based smart contracts in logistics and supply chains,

    M. A. Alqarni, S. H. Chauhdary, and S. Saleem, “Use of blockchain-based smart contracts in logistics and supply chains,”Electronics, vol. 12, no. 6, p. 1340, 2023

    work page 2023

  6. [6]

    An empirical study of smart contract decompilers,

    X. Liu, B. Hua, and Z. Pan, “An empirical study of smart contract decompilers,” inSANER, pp. 1–12, 2023

    work page 2023

  7. [7]

    Super-optimization of smart con- tracts,

    E. Albert, P. Gordillo, A. Hern ´andez-Cerezo, A. Rubio, and M. A. Schett, “Super-optimization of smart con- tracts,”TOSEM, vol. 31, no. 4, pp. 1–29, 2022

    work page 2022

  8. [8]

    A survey of compiler testing,

    J. Chen, J. Patra, M. Pradel, Y . Xiong, H. Zhang, D. Hao, and L. Zhang, “A survey of compiler testing,”CSUR, vol. 53, no. 1, pp. 1–36, 2020

    work page 2020

Show all 34 references
  1. [9]

    Differential testing: a new approach to change detection,

    R. B. Evans and A. Savoia, “Differential testing: a new approach to change detection,” inThe 6th Joint Meeting on European Software Engineering Conference and the ACM SIGSOFT Symposium on the F oundations of Software Engineering: Companion Papers, ESEC-FSE companion ’07, (New ...

    2007

  2. [10]

    Does the failing test execute a single or multiple faults? an approach to classifying failing tests,

    Z. Yu, C. Bai, and K.-Y . Cai, “Does the failing test execute a single or multiple faults? an approach to classifying failing tests,” inProceedings of the 37th Inter- national Conference on Software Engineering - V olume 1, ICSE ’15, p. 924–935, IEEE Press, 2015

    2015

  3. [11]

    Alleviating patch overfitting with automatic test generation: A study of feasibility and effectiveness for the nopol repair system,

    Z. Yu, M. Martinez, B. Danglot, T. Durieux, and M. Mon- perrus, “Alleviating patch overfitting with automatic test generation: A study of feasibility and effectiveness for the nopol repair system,”Empirical Softw. Engg., vol. 24, p. 33–67, feb 2019

    2019

  4. [12]

    A snowballing literature study on test amplification,

    B. Danglot, O. Vera-Perez, Z. Yu, A. Zaidman, M. Mon- perrus, and B. Baudry, “A snowballing literature study on test amplification,”Journal of Systems and Software, vol. 157, p. 110398, 2019

    2019

  5. [13]

    Mutation-oriented test data augmentation for gui software fault localization,

    Z. Yu, C. Bai, and K.-Y . Cai, “Mutation-oriented test data augmentation for gui software fault localization,” Information and Software Technology, vol. 55, no. 12, pp. 2076–2098, 2013

    2013

  6. [14]

    Boosting compiler testing by injecting real-world code,

    S. Li, T. Theodoridis, and Z. Su, “Boosting compiler testing by injecting real-world code,”PLDI, pp. 223–245

  7. [15]

    Taming compiler fuzzers,

    Y . Chen, A. Groce, C. Zhang, W.-K. Wong, X. Fern, E. Eide, and J. Regehr, “Taming compiler fuzzers,” in PLDI, pp. 197–208, 2013

    2013

  8. [16]

    Llm-based code generation method for golang compiler testing,

    Q. Gu, “Llm-based code generation method for golang compiler testing,” inESEC/FSE, pp. 2201–2203, 2023

    2023

  9. [17]

    Syntax-aware mutation for testing the solidity compiler,

    C. Mitropoulos, T. Sotiropoulos, S. Ioannidis, and D. Mitropoulos, “Syntax-aware mutation for testing the solidity compiler,” inESORICS, pp. 327–347, 2023

    2023

  10. [18]

    Differential testing solidity compiler through deep contract manipu- lation and mutation,

    Z. Tian, F. Wang, Y . Chen, and L. Chen, “Differential testing solidity compiler through deep contract manipu- lation and mutation,”Software Quality Journal, vol. 32, p. 765–790, Apr. 2024

    2024

  11. [19]

    Spectest: Specification-based compiler testing,

    R. Schumi and J. Sun, “Spectest: Specification-based compiler testing,” inF ASE 2021, pp. 269–291, 2021

    2021

  12. [20]

    Finding and understanding bugs in c compilers,

    X. Yang, Y . Chen, E. Eide, and J. Regehr, “Finding and understanding bugs in c compilers,” inPLDI, pp. 283– 294, 2011

    2011

  13. [21]

    Kubo: Precise and scalable detection of user-triggerable undefined behavior bugs in os kernel.,

    C. Liu, Y . Chen, and L. Lu, “Kubo: Precise and scalable detection of user-triggerable undefined behavior bugs in os kernel.,” inNDSS, 2021

    2021

  14. [22]

    Differential testing for software,

    W. M. McKeeman, “Differential testing for software,” Digital Technical Journal, vol. 10, no. 1, pp. 100–107

  15. [23]

    Differential testing: a new approach to change detection,

    R. B. Evans and A. Savoia, “Differential testing: a new approach to change detection,” inESEC/FSE, pp. 549– 552, 2007

    2007

  16. [24]

    Rustsmith: Random differential compiler testing for rust,

    M. Sharma, P. Yu, and A. F. Donaldson, “Rustsmith: Random differential compiler testing for rust,” inISSTA, pp. 1483–1486, 2023

    2023

  17. [25]

    Random testing for c and c++ compilers with yarpgen,

    V . Livinskii, D. Babokin, and J. Regehr, “Random testing for c and c++ compilers with yarpgen,”OOPSLA, pp. 1– 25, 2020

    2020

  18. [26]

    using-the-compiler: input-description

    Soliditylang, “using-the-compiler: input-description.” https://docs.soliditylang.org/en/latest/using-the-compiler. html#input-description, 2025

    2025

  19. [27]

    https://hardhat.org/docs, 2025

    hardhat. https://hardhat.org/docs, 2025. [Online] ac- cessed 2025-02-24

    2025

  20. [28]

    Compiler testing using template java programs,

    Z. Zang, N. Wiatrek, M. Gligoric, and A. Shi, “Compiler testing using template java programs,” inASE, pp. 1–13

  21. [29]

    Compiler optimiza- tion testing based on optimization-guided equivalence transformations,

    J. Wu, J. Zheng, Z. Yang, and Z. Yu, “Compiler optimiza- tion testing based on optimization-guided equivalence transformations,” inCompanion Proceedings of the 33rd ACM International Conference on the F oundations of Software Engineering, FSE, 2025

    2025

  22. [30]

    Test-case reduction and deduplication almost for free with transformation- based compiler testing,

    A. F. Donaldson, P. Thomson, V . Teliman, S. Milizia, A. P. Maselco, and A. Karpi ´nski, “Test-case reduction and deduplication almost for free with transformation- based compiler testing,” inPLDI, pp. 1017–1032, 2021

    2021

  23. [31]

    Finding typing com- piler bugs,

    S. Chaliasos, T. Sotiropoulos, D. Spinellis, A. Gervais, B. Livshits, and D. Mitropoulos, “Finding typing com- piler bugs,” inPLDI, pp. 183–198, 2022

    2022

  24. [32]

    Nnsmith: Generating diverse and valid test cases for deep learning compilers,

    J. Liu, J. Lin, C. Tan, A. Panda, and L. Zhang, “Nnsmith: Generating diverse and valid test cases for deep learning compilers,” inASPLOS, pp. 530–543, 2023

    2023

  25. [33]

    A new methodology for generating test cases for a programming language compiler,

    D. M. Berry, “A new methodology for generating test cases for a programming language compiler,”ACM Sig- plan Notices, vol. 18, no. 2, pp. 46–56, 1983

    1983

  26. [34]

    Orthogonal latin squares: an application of experiment design to compiler testing,

    R. Mandl, “Orthogonal latin squares: an application of experiment design to compiler testing,”Communications of the ACM, vol. 28, no. 10, pp. 1054–1058, 1985

    1985

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.