Pith. sign in

REVIEW 4 major objections 5 minor 74 references

Distinguishability-guided Test Program Generation for WebAssembly Runtime Performance Testing

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read WarpGen claims that guiding test-program generation by a runtime-ratio deviation score efficiently yields high-quality WebAssembly performance tests, uncovering seven new runtime issues.

desk verdict WarpGen is a genuine first step for Wasm runtime performance test generation with seven confirmed issues, but its evaluation leans on the very dist score it optimizes, so the real-world issues are what carry the paper. read the letter →

arxiv 2412.20100 v1 pith:XKIRKFDR submitted 2024-12-28 cs.SE

Shuyao Jiang , Ruiying Zeng , Yangfan Zhou , Michael R. Lyu This is my paper · ORCID
classification cs.SE
keywords WebAssemblyperformancetestingtestprogramgenerationdistinguishabilitydifferentialruntimeoperatorextraction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that WebAssembly runtime performance testing suffers from a shortage of test programs that actually trigger performance issues, and that automatically generated programs can fill that gap if generation is guided by the right quality signal. It presents WarpGen, which reuses code snippets from previously known issue-triggering programs, inserts them into random C seed programs, and keeps the generated programs whose execution-time ratios across runtimes deviate most from a fitted oracle ratio. The paper claims WarpGen reaches near-optimal test quality within about 140 generated programs, outperforming random generation and an unguided variant, and that it uncovered seven previously unknown performance issues across Wasmtime, Wasmer, and WAMR. A sympathetic reader would take this as evidence that performance-test generation for Wasm runtimes is feasible and that a ratio-deviation score is a workable quality oracle.

What carries the argument

The load-bearing object is the distinguishability (dist score), defined as the Euclidean distance between the normalized vector of a test program's execution time on the tested Wasm runtimes and the normalized vector of the oracle ratio, where the oracle ratio is fitted as the average execution-time ratio of the seed programs. A program whose time ratio deviates far from the oracle ratio is judged able to distinguish abnormal performance of some runtime, and WarpGen uses this score to maintain a top-N set of distinguishable programs, to decide which synthesized programs enter the operator pool, and to prune operators via a penalty mechanism that removes an operator after M consecutive failures to improve the top-N scores.

What would settle it

Compile a diverse set of correctness-checked C programs with no known performance issues, run them on the same four runtimes in AOT mode, and compute their time-ratio vectors; if the spread of these ratios among known-good programs routinely reaches the deviations WarpGen classifies as distinguishable, the dist score cannot separate real issues from normal variance.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that the distinguishability score—the Euclidean distance between a test program's L1-normalized execution-time ratio vector across several Wasm runtimes and the corresponding normalized oracle-ratio vector—can serve as an automatic quality indicator that guides test program synthesis toward issue-triggering programs. WarpGen operationalizes this by extracting block-level operators from historical abnormal programs reported by WarpDiff, inserting them into Csmith-generated seed programs while resolving variable dependencies through pre/post contexts, and iteratively updating its operator pool with operators from newly distinguishable programs. The paper reports that WarpGen's top-20 dist score rises quickly to a near-optimal plateau, that its final scores are about four times those of random Csmith programs, and that manual analysis of the top distinguishable programs yielded seven confirmed, previously unknown performance issues in three runtimes.

Load-bearing premise

The method assumes that, under normal conditions, the same test program's execution time on different Wasm runtimes keeps a stable ratio, so any large deviation from the average ratio marks a genuine performance anomaly rather than ordinary variation across programs.

Editorial extensions

If this is right

  • WarpGen can generate high-quality test programs faster than random Csmith programs and faster than an unguided version, with the top-20 dist score plateauing after roughly 140 generated programs.
  • Using WarpGen, seven previously unknown performance issues were found in three Wasm runtimes, four of which have already been fixed.
  • Operators extracted from historical issue-triggering programs, and later from newly distinguishable programs, both contribute to detecting new issues.
  • A single inserted operator can trigger issues in different runtimes when placed in different seed contexts, as with the FP-arithmetic case in Wasmtime and WAMR.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The oracle-ratio assumption could be checked more directly: the paper fits the oracle only from seed programs, so if the seed pool is not representative, the oracle itself may shift and change which programs look distinguishable.
  • A testable extension is to run the dist-score-guided loop with different seed pools and see whether the identified issues and top programs remain stable.
  • The dist score as defined only captures relative timing across runtimes; it would not detect a performance regression that hits all runtimes proportionally, so a complementary absolute-time oracle would be needed for full coverage.
  • The paper's early-stop behavior (the operator pool emptied at 436 programs) suggests the penalty mechanism aggressively prunes; a sensitivity analysis on the penalty threshold M would show whether this speed comes at the cost of missing longer-tail issues.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper presents WarpGen, a test program generation approach for WebAssembly (Wasm) runtime performance testing. WarpGen extracts code snippets ('operators') from previously known issue-triggering test programs, synthesizes new C programs by inserting operators into seed programs, and uses a 'distinguishability' score (dist score) to guide an iterative search for programs whose execution-time ratios across four Wasm runtimes deviate from an oracle ratio. The authors evaluate WarpGen against random Csmith-generated programs and a non-guided variant, and report that WarpGen identifies seven previously unknown performance issues in Wasmer, Wasmtime, and WAMR, four of which have been fixed. The central claims are that WarpGen generates high-quality test programs more efficiently than baselines and that the distinguishability-guided design is effective.

Significance. If the validity of the dist score as a proxy for issue-triggering capability were established, WarpGen would be a useful contribution to a relatively underexplored area: the paper is well-motivated, the tool is implemented and released, and the seven developer-confirmed issues are concrete evidence that the approach can find real problems. The history-driven operator extraction with context handling is technically sound. However, the evaluation's dependence on the optimization target reduces the evidential weight of the efficiency and effectiveness claims; the actual bug-finding results are limited to a small sample with no false-positive count or baseline comparison. With the additional validation, the paper could be a solid contribution.

major comments (4)
  1. [Section III-C and Section IV-A (Parameters)] The dist score is defined as the Euclidean distance between the normalized execution-time-ratio vector and the normalized oracle-ratio vector, where the oracle ratio is the average execution-time ratio of the 100 seed programs. RQ1 (Table II) and RQ2 (Figure 4) then evaluate WarpGen's outputs using this same dist score. Because WarpGen's iteration process explicitly selects programs with high dist score, these experiments amount to checking whether WarpGen achieves a high score in the objective it was designed to maximize. They do not, by themselves, establish that the generated programs are of higher quality in the sense of triggering real performance issues. The only non-circular evidence is the seven confirmed issues in RQ3, which need to be supplemented with precision/false-positive data and a same-criterion comparison with baselines.
  2. [Section IV-D, Table III] The paper reports that the top 20 distinguishable programs were analyzed and seven performance issues were confirmed. The paper does not state how many of the top-20 programs were examined but did not correspond to real issues, nor does it describe the verification procedure for the 13 non-confirmed programs. Reporting the false-positive count (and the criteria used to classify a program as not issue-triggering) is essential to judge whether the dist score actually reduces manual effort. In addition, the paper should run the same confirmation procedure on top-20 programs produced by Csmith and WarpGen-base to support the claim that WarpGen is more efficient at finding real issues; the dist-score comparison in Figure 4 cannot substitute for this.
  3. [Section II-B and Section III-C] The oracle-ratio stability assumption is load-bearing but unverified. The paper inherits from WarpDiff the premise that, normally, the execution-time ratio of a given program across runtimes is stable. However, WarpGen intentionally creates programs with heterogeneous operator compositions and variable substitutions; such programs may legitimately have cross-runtime ratios far from the seed-program average without any performance bug. The seven confirmed issues show a positive correlation, but no evidence is provided about the rate of high-dist programs that do not correspond to bugs. The authors should provide a control experiment: measure dist scores for a set of programs that are known not to be issue-triggering (e.g., a fresh set of Csmith programs or the 123 LLVM Test Suite programs) and show that high dist scores are rare in that set, or demonstrate that the top-20 programs' dist scores are outliers relative to such a control distribution.
  4. [Section IV-B and Section IV-C (Execution methodology)] The paper does not describe how execution times are measured. It reports dist scores without any information about the number of repetitions, whether means/medians are used, or how measurement noise is handled. Since the dist score is central to all three research questions, the absence of a measurement methodology makes it impossible to assess whether the reported differences between approaches are within noise. At minimum, the authors should state the repetition count and aggregation method, and report variances or confidence intervals for the key results in Table II and Figure 4.
minor comments (5)
  1. [Section IV-C] The typo 'Csimth' should be corrected to 'Csmith'.
  2. [Contributions] The paper claims 'the first study on test program generation for Wasm runtime performance testing'; given that related work cites Wapplique (bytecode mutation testing of Wasm runtimes), the authors should clarify the distinction or temper the 'first' claim.
  3. [Section IV-D] The term 'deviation degree' is used but not formally defined; provide the formula used to compute it.
  4. [Section IV-B] The paper reports that the operator pool became empty after 436 programs; it would be useful to report the initial number of operators and the number of operators extracted during iterations to understand the search dynamics.
  5. [Data availability] The Figshare link is given, but the availability statement could specify the exact version and contents (e.g., scripts, raw data, issue reports) to support reproducibility.

Circularity Check

2 steps flagged · score 4.0 of 10

Dist score is both the optimization objective and the main evaluation metric; the oracle-ratio premise is inherited from same-author WarpDiff, leaving RQ3's seven confirmed issues as the only independent grounding.

  1. fitted input called prediction [Section IV-A (Parameters) and Section IV-B (RQ1)]
    "we calculated the oracle ratio based on the average execution time ratio of the seed programs, which represents the normal performance of the tested Wasm runtimes. ... To evaluate the efficiency of WarpGen to generate high-quality test programs, we collected the statistics of dist score for each generated test program during the iteration process of WarpGen."

    The oracle ratio is fitted as the average execution-time ratio of the seed programs, and the dist score is defined as the distance from that fitted oracle. The paper then equates high dist score with high-quality, issue-triggering programs and uses the dist score as the main evidence in RQ1/RQ2. Because WarpGen's iteration loop keeps top-N dist programs and extracts new operators from exactly those programs, it is engineered to raise the dist score; comparing it with non-guided baselines on the same dist score therefore partly measures the algorithm against its own objective. The only independent evidence that high dist corresponds to real issues is the seven developer-confirmed issues in RQ3, which are not used as the RQ1/RQ2 quality metric.

  2. self citation load bearing [Section III (Approach, second paragraph) and Section IV-A (Initial Operators)]
    "To formalize it, we draw inspiration from the idea of WarpDiff, that is, the execution time of the same test case on different Wasm runtimes should follow a stable ratio (i.e., oracle ratio). ... we collected the top 20 abnormal test programs reported by WarpDiff for operator extraction, since they are currently the only source related to Wasm runtime performance issues."

    WarpDiff (reference [17]) is prior work by the same authors (Jiang, Zeng, Zhou, Lyu). The stability-of-oracle-ratio assumption is the load-bearing premise of the dist score: without it, a large distance from the seed-fitted oracle is not evidence of an abnormal performance issue. The paper adopts this premise and the historical issue-triggering programs from its own earlier paper, rather than independently re-deriving or testing the assumption. The RQ3 confirmation locates issues using the same deviation-from-oracle method, so it does not provide a fully independent validation of the premise for the space of generated programs.

full rationale

WarpGen's derivation chain is: high-quality programs are those that trigger performance issues; the paper operationalizes this as a high dist score, defined as Euclidean distance from an oracle ratio; the oracle ratio is fitted as the average execution-time ratio of the 100 Csmith seed programs; generation is guided by keeping top-N dist programs and re-extracting operators from them; and RQ1/RQ2 evaluate the approach by reporting dist-score statistics. This chain is partially circular because the optimization objective and the primary evaluation metric are the same fitted quantity, so higher dist scores for WarpGen relative to non-guided baselines are in part guaranteed by the selection mechanism. The independent content comes from RQ3: seven previously unknown performance issues confirmed by developers, which shows that at least some top-dist programs correspond to real bugs. However, RQ3 uses the same deviation-from-oracle method to select and locate those issues, reports no false-positive count, and does not apply the same developer-confirmation criterion to the baseline approaches. The oracle-ratio stability premise is inherited from WarpDiff, a same-author publication, without independent justification in this paper. These factors support a moderate score of 4 rather than 0 or 2, but they do not make the entire contribution definitionally circular because the developer confirmations are external to the dist-score construction. The threats-to-validity section discusses parameter choices but does not acknowledge this objective/metric overlap.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central machinery rests on the oracle-ratio assumption inherited from WarpDiff, the history-driven heuristic that old issue-triggering snippets help find new issues, and a fitted baseline from seed programs. Tuning parameters N, M, and k influence which programs are generated and reported. No new physical entities are introduced; the only invented construct is the distinguishability indicator.

free parameters (4)
  • oracle ratio vector = not numerically reported; average of seed program execution time ratios
    Defines normal behavior; dist score is the distance from this vector. It is fitted from the seed pool rather than independently established.
  • N (top distinguishable set size) = 20
    Chosen from WarpDiff's reported data in Section IV-A; controls the threshold for updating the operator pool and the reported top scores.
  • M (operator penalty threshold) = 5
    Selected based on the authors' pre-experiments to balance efficiency and quality, as stated in Section IV-A and Section V-B.
  • k (generation budget) = 1000, but the experiment stopped early at 436 programs
    Stopping condition for the iteration loop; the reported efficiency claims come from the 436-program run rather than the full budget.
assumptions (4)
  • domain assumption Execution time ratios between runtimes are stable under normal circumstances.
    Used in Section II-B from WarpDiff and Section III-C to define distinguishability; if false, deviation from the oracle ratio does not indicate a performance issue.
  • domain assumption Code snippets from historical issue-triggering programs help detect new issues.
    Stated as the first design insight in Section III; adopted from compiler and JVM testing, not independently proven for Wasm.
  • domain assumption The oracle ratio computed from seed programs is representative of normal performance.
    Section IV-A calculates the oracle from Csmith-generated seeds only; non-representative seeds would bias the baseline.
  • domain assumption The runtime with the largest deviation degree is the runtime responsible for the anomaly.
    Section IV-D uses this to assign each distinguishable program to one issue-related runtime; it assumes deviations are not due to noise or multiple runtimes.
invented entities (1)
  • distinguishability (dist score) independent evidence
    purpose: Quality indicator for generated test programs; guides operator and seed selection and defines what counts as a high-quality program.
    The paper ties high dist score to developer-confirmed performance issues, which are external to the metric. However, the metric is defined relative to a fitted oracle ratio, so it is not independently calibrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Distinguishability-guided Test Program Generation for WebAssembly Runtime Performance Testing." pith.science (2026). https://pith.science/paper/XKIRKFDR

@misc{pith2026241220100,
  author       = {Pith},
  title        = {Pith review of: Distinguishability-guided Test Program Generation for WebAssembly Runtime Performance Testing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XKIRKFDR}},
  note         = {Machine review of arXiv:2412.20100}
}
read the original abstract

WebAssembly (Wasm) is a binary instruction format designed as a portable compilation target, which has been widely used on both the web and server sides in recent years. As high performance is a critical design goal of Wasm, it is essential to conduct performance testing for Wasm runtimes. However, existing research on Wasm runtime performance testing still suffers from insufficient high-quality test programs. To solve this problem, we propose a novel test program generation approach WarpGen. It first extracts code snippets from historical issue-triggering test programs as initial operators, then inserts an operator into a seed program to synthesize a new test program. To verify the quality of generated programs, we propose an indicator called distinguishability, which refers to the ability of a test program to distinguish abnormal performance of specific Wasm runtimes. We apply WarpGen for performance testing on four Wasm runtimes and verify its effectiveness compared with baseline approaches. In particular, WarpGen has identified seven new performance issues in three Wasm runtimes.

Figures

Figures reproduced from arXiv: 2412.20100 by the authors.

Figure 1
Figure 1. The workflow of Wasm compilation and execution. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The framework of WarpGen. called dist score. WarpGen uses the dist score to guide the test program generation process [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. A looping operator and its pre/post-contexts. The type and positions of the variables are omitted. and looping statements are treated as sequential operators, and the rest are treated as mixed operators. As mentioned above, a critical challenge in program synthe￾sis is to ensure the syntax validity and insertion validity of the new program. To achieve this goal, during operator extraction, WarpGen records the follow… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of top 20 dist score from different approaches. can generate high-quality test programs for Wasm runtime performance testing with high efficiency. C. RQ2: Effectiveness of Guidance To further study the effectiveness of WarpGen, especially the distinguishabil…
Figure 5
Figure 5. Figure 5: Test program related to Issue #7731. design insight of WarpGen, i.e., the historical issue-triggering test programs can help detect new issues. Issue #7732 is related to the improper optimization when accessing the pointers to constant. In this case, the inserted opera…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

74 extracted references · 69 canonical work pages

  1. [1]

    Bringing the web up to speed with webassembly,

    A. Haas, A. Rossberg, D. L. Schuff, B. L. Titzer, M. Holman, D. Gohman, L. Wagner, A. Zakai, and J. Bastien, “Bringing the web up to speed with webassembly,” in Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation , 2017, pp. 185–200

    work page 2017

  2. [2]

    That’sa tough call: Studying the challenges of call graph construction for webassembly,

    D. Lehmann, M. Thalakottur, F. Tip, and M. Pradel, “That’sa tough call: Studying the challenges of call graph construction for webassembly,” in Symposium on Software Testing and Analysis (ISSTA’23) , 2023

    work page 2023

  3. [3]

    Finding the dwarf: recovering precise types from webassembly binaries,

    D. Lehmann and M. Pradel, “Finding the dwarf: recovering precise types from webassembly binaries,” in Proceedings of the 43rd ACM SIGPLAN International Conference on Programming Language Design and Implementation, 2022, pp. 410–425

    work page 2022

  4. [4]

    Everything old is new again: Binary security of {WebAssembly},

    D. Lehmann, J. Kinder, and M. Pradel, “Everything old is new again: Binary security of {WebAssembly},” in 29th USENIX Security Sympo- sium (USENIX Security 20) , 2020, pp. 217–234

    work page 2020

  5. [5]

    Wasabi: A framework for dynamically ana- lyzing webassembly,

    D. Lehmann and M. Pradel, “Wasabi: A framework for dynamically ana- lyzing webassembly,” in Proceedings of the Twenty-Fourth International Conference on Architectural Support for Programming Languages and Operating Systems, 2019, pp. 1045–1058

    work page 2019

  6. [6]

    Automated webassembly function purpose identification with semantics-aware analysis,

    A. Romano and W. Wang, “Automated webassembly function purpose identification with semantics-aware analysis,” inProceedings of the ACM Web Conference 2023, 2023, pp. 2885–2894

    work page 2023

  7. [7]

    A webassembly milestone: Experimental support in multiple browsers,

    L. Wagner, “A webassembly milestone: Experimental support in multiple browsers,” Mozilla Hacks (14 March 2016). , 2017

    work page 2016

  8. [8]

    Faasm: Lightweight isolation for efficient stateful serverless computing,

    S. Shillaker and P. Pietzuch, “Faasm: Lightweight isolation for efficient stateful serverless computing,” in 2020 USENIX Annual Technical Conference (USENIX ATC 20) , 2020, pp. 419–433

    work page 2020

Show all 74 references
  1. [9]

    Pushing serverless to the edge with webassembly runtimes,

    P. Gackstatter, P. A. Frangoudis, and S. Dustdar, “Pushing serverless to the edge with webassembly runtimes,” in 2022 22nd IEEE International Symposium on Cluster, Cloud and Internet Computing (CCGrid). IEEE, 2022, pp. 140–149

    2022

  2. [10]

    Sledge: A serverless-first, light-weight wasm runtime for the edge,

    P. K. Gadepalli, S. McBride, G. Peach, L. Cherkasova, and G. Parmer, “Sledge: A serverless-first, light-weight wasm runtime for the edge,” in Proceedings of the 21st International Middleware Conference, 2020, pp. 265–279

    2020

  3. [11]

    Warduino: a dynamic webassem- bly virtual machine for programming microcontrollers,

    R. Gurdeep Singh and C. Scholliers, “Warduino: a dynamic webassem- bly virtual machine for programming microcontrollers,” in Proceedings of the 16th ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes , 2019, pp. 27–36

    2019

  4. [12]

    Femto-containers: Devops on micro- controllers with lightweight virtualization & isolation for iot software modules,

    K. Zandberg and E. Baccelli, “Femto-containers: Devops on micro- controllers with lightweight virtualization & isolation for iot software modules,” arXiv preprint arXiv:2106.12553 , 2021

    2021 arXiv

  5. [13]

    Vm matters: A comparison of wasm vms and evms in the performance of blockchain smart contracts,

    S. Zheng, H. Wang, L. Wu, G. Huang, and X. Liu, “Vm matters: A comparison of wasm vms and evms in the performance of blockchain smart contracts,” arXiv preprint arXiv:2012.01032 , 2020

    2012 arXiv

  6. [14]

    Wana: Symbolic execution of wasm bytecode for cross-platform smart contract vulnerability detection,

    D. Wang, B. Jiang, and W. Chan, “Wana: Symbolic execution of wasm bytecode for cross-platform smart contract vulnerability detection,” arXiv preprint arXiv:2007.15510 , 2020

    2007 arXiv

  7. [15]

    Wasai: uncovering vulnerabilities in wasm smart contracts,

    W. Chen, Z. Sun, H. Wang, X. Luo, H. Cai, and L. Wu, “Wasai: uncovering vulnerabilities in wasm smart contracts,” in Proceedings of the 31st ACM SIGSOFT International Symposium on Software Testing and Analysis, 2022, pp. 703–715

    2022

  8. [16]

    Wasmedge,

    “Wasmedge,” https://github.com/WasmEdge/WasmEdge, 2024

    2024

  9. [17]

    Revealing performance issues in server-side webassembly runtimes via differential testing,

    S. Jiang, R. Zeng, Z. Rao, J. Gu, Y . Zhou, and M. R. Lyu, “Revealing performance issues in server-side webassembly runtimes via differential testing,” in Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 661–672

    2023

  10. [18]

    Secure & lightweight microservice with a database backend,

    “Secure & lightweight microservice with a database backend,” https: //github.com/second-state/microservice-rust-mysql, 2024

    2024

  11. [19]

    Llvm test suite,

    “Llvm test suite,” https://github.com/llvm/llvm-test-suite, 2024

    2024

  12. [20]

    Finding and understanding bugs in c compilers,

    X. Yang, Y . Chen, E. Eide, and J. Regehr, “Finding and understanding bugs in c compilers,” in Proceedings of the 32nd ACM SIGPLAN Conference on Programming Language Design and Implementation , 2011, pp. 283–294

    2011

  13. [21]

    History- guided configuration diversification for compiler test-program genera- tion,

    J. Chen, G. Wang, D. Hao, Y . Xiong, H. Zhang, and L. Zhang, “History- guided configuration diversification for compiler test-program genera- tion,” in 2019 34th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2019, pp. 305–316

    2019

  14. [22]

    Configuring test generators using bug reports: a case study of gcc compiler and csmith,

    M. R. I. Rabin and M. A. Alipour, “Configuring test generators using bug reports: a case study of gcc compiler and csmith,” in Proceedings of the 36th Annual ACM Symposium on Applied Computing , 2021, pp. 1750–1758

    2021

  15. [23]

    History-driven test program synthesis for jvm testing,

    Y . Zhao, Z. Wang, J. Chen, M. Liu, M. Wu, Y . Zhang, and L. Zhang, “History-driven test program synthesis for jvm testing,” in Proceedings of the 44th International Conference on Software Engineering , 2022, pp. 1133–1144

    2022

  16. [24]

    “Wasmer,” https://github.com/wasmerio/wasmer, 2024

    2024

  17. [25]

    Wasmtime,

    “Wasmtime,” https://github.com/bytecodealliance/wasmtime, 2024

    2024

  18. [26]

    Webassembly micro runtime (wamr),

    “Webassembly micro runtime (wamr),” https://github.com/ bytecodealliance/wasm-micro-runtime, 2024

    2024

  19. [27]

    V8 javascript engine,

    “V8 javascript engine,” https://v8.dev/, 2024

    2024

  20. [28]

    Spidermonkey javascript/webassembly engine,

    “Spidermonkey javascript/webassembly engine,” https://spidermonkey. dev/, 2024

    2024

  21. [29]

    Standardizing wasi: A system interface to run webassembly outside the web,

    L. Clark, “Standardizing wasi: A system interface to run webassembly outside the web,” Mozilla Hacks–the Web developer blog , 2019

    2019

  22. [30]

    Awesome webassembly runtimes,

    “Awesome webassembly runtimes,” https://github.com/appcypher/ awesome-wasm-runtimes, 2024

    2024

  23. [31]

    Not so fast: Analyzing the performance of {WebAssembly} vs. native code,

    A. Jangda, B. Powers, E. D. Berger, and A. Guha, “Not so fast: Analyzing the performance of {WebAssembly} vs. native code,” in 2019 USENIX Annual Technical Conference (USENIX ATC 19) , 2019, pp. 107–120

    2019

  24. [32]

    Empowering web applications with webassembly: Are we there yet?

    W. Wang, “Empowering web applications with webassembly: Are we there yet?” in 2021 36th IEEE/ACM International Conference on Auto- mated Software Engineering (ASE) . IEEE, 2021, pp. 1301–1305

    2021

  25. [33]

    Understanding the performance of webassembly applications,

    Y . Yan, T. Tu, L. Zhao, Y . Zhou, and W. Wang, “Understanding the performance of webassembly applications,” in Proceedings of the 21st ACM Internet Measurement Conference , 2021, pp. 533–549

    2021

  26. [34]

    An evaluation of webassembly in non-web environments,

    B. Spies and M. Mock, “An evaluation of webassembly in non-web environments,” in 2021 XLVII Latin American Computing Conference (CLEI). IEEE, 2021, pp. 1–10

    2021

  27. [35]

    Llvm project,

    “Llvm project,” https://github.com/llvm/llvm-project, 2024

    2024

  28. [36]

    Emscripten: an llvm-to-javascript compiler,

    A. Zakai, “Emscripten: an llvm-to-javascript compiler,” in Proceedings of the ACM international conference companion on Object oriented programming systems languages and applications companion, 2011, pp. 301–312

    2011

  29. [37]

    Accelerate javascript applications by cross- compiling to webassembly,

    M. Reiser and L. Bl ¨aser, “Accelerate javascript applications by cross- compiling to webassembly,” in Proceedings of the 9th ACM SIGPLAN International Workshop on Virtual Machines and Intermediate Lan- guages, 2017, pp. 10–17

    2017

  30. [38]

    An empirical study of real- world webassembly binaries: Security, languages, use cases,

    A. Hilbig, D. Lehmann, and M. Pradel, “An empirical study of real- world webassembly binaries: Security, languages, use cases,” in Pro- ceedings of the Web Conference 2021 , 2021, pp. 2696–2708

    2021

  31. [39]

    Wobfuscator: Obfuscating javascript malware via opportunistic translation to we- bassembly,

    A. Romano, D. Lehmann, M. Pradel, and W. Wang, “Wobfuscator: Obfuscating javascript malware via opportunistic translation to we- bassembly,” in 2022 IEEE Symposium on Security and Privacy (SP) . IEEE, 2022, pp. 1574–1589

    2022

  32. [40]

    Evaluating webassembly enabled serverless approach for edge computing,

    P. Mendki, “Evaluating webassembly enabled serverless approach for edge computing,” in 2020 IEEE Cloud Summit . IEEE, 2020, pp. 161– 166

    2020

  33. [41]

    Webassembly modules as lightweight containers for liquid iot applications,

    N. M ¨akitalo, T. Mikkonen, C. Pautasso, V . Bankowski, P. Daubaris, R. Mikkola, and O. Beletski, “Webassembly modules as lightweight containers for liquid iot applications,” in Web Engineering: 21st Inter- national Conference, ICWE 2021, Biarritz, France, May 18–21, 2021, Pro...

    2021

  34. [42]

    Evaluating webassembly for orchestrated deployment of serverless functions,

    V . Kjorveziroski, S. Filiposka, and A. Mishev, “Evaluating webassembly for orchestrated deployment of serverless functions,” in 2022 30th Telecommunications Forum (TELFOR). IEEE, 2022, pp. 1–4

    2022

  35. [43]

    Nomad: Cross-platform compu- tational offloading and migration in femtoclouds using webassembly,

    M. Nurul-Hoque and K. A. Harras, “Nomad: Cross-platform compu- tational offloading and migration in femtoclouds using webassembly,” in 2021 IEEE International Conference on Cloud Engineering (IC2E) . IEEE, 2021, pp. 168–178

    2021

  36. [44]

    Webassembly ver- sus javascript: Energy and runtime performance,

    J. De Macedo, R. Abreu, R. Pereira, and J. Saraiva, “Webassembly ver- sus javascript: Energy and runtime performance,” in 2022 International Conference on ICT for Sustainability (ICT4S) . IEEE, 2022, pp. 24–34

    2022

  37. [45]

    On the runtime and energy performance of webassembly: Is webassembly superior to javascript yet?

    ——, “On the runtime and energy performance of webassembly: Is webassembly superior to javascript yet?” in 2021 36th IEEE/ACM In- ternational Conference on Automated Software Engineering Workshops (ASEW). IEEE, 2021, pp. 255–262

    2021

  38. [46]

    When function inlining meets webassembly: Counterintuitive impacts on runtime performance,

    A. Romano and W. Wang, “When function inlining meets webassembly: Counterintuitive impacts on runtime performance,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE) . ACM, 2023

    2023

  39. [47]

    A comprehensive study of webassembly runtime bugs,

    Y . Wang, Z. Zhou, Z. Ren, D. Liu, and H. Jiang, “A comprehensive study of webassembly runtime bugs,” in 2023 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER) . IEEE, 2023, pp. 355–366

    2023

  40. [48]

    Characterizing and detecting webassembly runtime bugs,

    Y . Zhang, S. Cao, H. Wang, Z. Chen, X. Luo, D. Mu, Y . Ma, G. Huang, and X. Liu, “Characterizing and detecting webassembly runtime bugs,” arXiv preprint arXiv:2301.12102 , 2023

    2023 arXiv

  41. [49]

    Exploring missed optimizations in webassembly optimizers,

    Z. Liu, D. Xiao, Z. Li, S. Wang, and W. Meng, “Exploring missed optimizations in webassembly optimizers,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, 2023, pp. 436–448

    2023

  42. [50]

    An empirical study of bugs in webassembly compilers,

    A. Romano, X. Liu, Y . Kwon, and W. Wang, “An empirical study of bugs in webassembly compilers,” in 2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2021, pp. 42–54

    2021

  43. [51]

    Compiler test case generation methods: a survey and assessment,

    A. S. Boujarwah and K. Saleh, “Compiler test case generation methods: a survey and assessment,” Information and software technology, vol. 39, no. 9, pp. 617–625, 1997

    1997

  44. [52]

    Learning to prioritize test programs for compiler testing,

    J. Chen, Y . Bai, D. Hao, Y . Xiong, H. Zhang, and B. Xie, “Learning to prioritize test programs for compiler testing,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE). IEEE, 2017, pp. 700–711

    2017

  45. [53]

    Compiler bug isolation via effective witness test program generation,

    J. Chen, J. Han, P. Sun, L. Zhang, D. Hao, and L. Zhang, “Compiler bug isolation via effective witness test program generation,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineeri...

    2019

  46. [54]

    A survey of compiler testing,

    J. Chen, J. Patra, M. Pradel, Y . Xiong, H. Zhang, D. Hao, and L. Zhang, “A survey of compiler testing,” ACM Computing Surveys (CSUR) , vol. 53, no. 1, pp. 1–36, 2020

    2020

  47. [55]

    Jitfuzz: Coverage-guided fuzzing for jvm just-in-time compilers,

    M. Wu, M. Lu, H. Cui, J. Chen, Y . Zhang, and L. Zhang, “Jitfuzz: Coverage-guided fuzzing for jvm just-in-time compilers,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 56–68

    2023

  48. [56]

    Remgen: Remanufacturing a random program generator for compiler testing,

    H. Tu, H. Jiang, X. Li, Z. Ren, Z. Zhou, and L. Jiang, “Remgen: Remanufacturing a random program generator for compiler testing,” in 2022 IEEE 33rd International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2022, pp. 529–540

    2022

  49. [57]

    Coverage-directed differential testing of jvm implementations,

    Y . Chen, T. Su, C. Sun, Z. Su, and J. Zhao, “Coverage-directed differential testing of jvm implementations,” in proceedings of the 37th ACM SIGPLAN Conference on Programming Language Design and Implementation, 2016, pp. 85–99

    2016

  50. [58]

    Deep differential testing of jvm implemen- tations,

    Y . Chen, T. Su, and Z. Su, “Deep differential testing of jvm implemen- tations,” in 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 2019, pp. 1257–1268

    2019

  51. [59]

    Jvm fuzzing for jit-induced side- channel detection,

    T. Brennan, S. Saha, and T. Bultan, “Jvm fuzzing for jit-induced side- channel detection,” in Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering , 2020, pp. 1011–1023

    2020

  52. [60]

    Vectorizing program ingredients for better jvm testing,

    T. Gao, J. Chen, Y . Zhao, Y . Zhang, and L. Zhang, “Vectorizing program ingredients for better jvm testing,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , 2023, pp. 526–537

    2023

  53. [61]

    Random testing for c and c++ compilers with yarpgen,

    V . Livinskii, D. Babokin, and J. Regehr, “Random testing for c and c++ compilers with yarpgen,” Proceedings of the ACM on Programming Languages, vol. 4, no. OOPSLA, pp. 1–25, 2020

    2020

  54. [62]

    Generating focused random tests using directed swarm testing,

    M. A. Alipour, A. Groce, R. Gopinath, and A. Christi, “Generating focused random tests using directed swarm testing,” in Proceedings of the 25th International Symposium on Software Testing and Analysis , 2016, pp. 70–81

    2016

  55. [63]

    Swarm testing,

    A. Groce, C. Zhang, E. Eide, Y . Chen, and J. Regehr, “Swarm testing,” in Proceedings of the 2012 International Symposium on Software Testing and Analysis, 2012, pp. 78–88

    2012

  56. [64]

    Compiler validation via equivalence modulo inputs,

    V . Le, M. Afshari, and Z. Su, “Compiler validation via equivalence modulo inputs,” in Proceedings of the 35th ACM SIGPLAN Conference on Programming Language Design and Implementation, 2014, pp. 216– 226

    2014

  57. [65]

    Finding deep compiler bugs via guided stochastic program mutation,

    V . Le, C. Sun, and Z. Su, “Finding deep compiler bugs via guided stochastic program mutation,” in Proceedings of the 2015 ACM SIG- PLAN International Conference on Object-Oriented Programming, Sys- tems, Languages, and Applications , 2015, pp. 386–399

    2015

  58. [66]

    Finding compiler bugs via live code mutation,

    C. Sun, V . Le, and Z. Su, “Finding compiler bugs via live code mutation,” in Proceedings of the 2016 ACM SIGPLAN international conference on object-oriented programming, systems, languages, and applications , 2016, pp. 849–863

    2016

  59. [67]

    Many-core com- piler fuzzing,

    C. Lidbury, A. Lascu, N. Chong, and A. F. Donaldson, “Many-core com- piler fuzzing,” in Proceedings of the 36th ACM SIGPLAN Conference on Programming Language Design and Implementation , 2015, pp. 65–76

    2015

  60. [68]

    Metamorphic testing for (graphics) compilers,

    A. F. Donaldson and A. Lascu, “Metamorphic testing for (graphics) compilers,” in Proceedings of the 1st international workshop on meta- morphic testing, 2016, pp. 44–47

    2016

  61. [69]

    Compiler fuzzing through deep learning,

    C. Cummins, P. Petoumenos, A. Murray, and H. Leather, “Compiler fuzzing through deep learning,” in Proceedings of the 27th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2018, pp. 95–105

    2018

  62. [70]

    Deepfuzz: Automatic generation of syntax valid c programs for fuzz testing,

    X. Liu, X. Li, R. Prajapati, and D. Wu, “Deepfuzz: Automatic generation of syntax valid c programs for fuzz testing,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 33, no. 01, 2019, pp. 1044– 1051

    2019

  63. [71]

    Wafl: Binary-only webassembly fuzzing with fast snapshots,

    K. Haßler and D. Maier, “Wafl: Binary-only webassembly fuzzing with fast snapshots,” in Reversing and Offensive-oriented Trends Symposium, 2021, pp. 23–30

    2021

  64. [72]

    Fuzzm: Finding memory bugs through binary-only instrumentation and fuzzing of webassembly,

    D. Lehmann, M. T. Torp, and M. Pradel, “Fuzzm: Finding memory bugs through binary-only instrumentation and fuzzing of webassembly,”arXiv preprint arXiv:2110.15433, 2021

    2021

  65. [73]

    Wadiff: A differential testing framework for webassembly runtimes,

    S. Zhou, M. Jiang, W. Chen, H. Zhou, H. Wang, and X. Luo, “Wadiff: A differential testing framework for webassembly runtimes,” in Pro- ceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering (ASE) . IEEE, 2023, pp. 939–950

    2023

  66. [74]

    Wapplique: Testing webassembly runtime via execution context-aware bytecode mutation,

    W. Zhao, R. Zeng, and Y . Zhou, “Wapplique: Testing webassembly runtime via execution context-aware bytecode mutation,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA) , 2024, pp. 1035–1047

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.