Mull it over: mutation testing based on LLVM

Alex Denisov; Stanislav Pankevich

REVIEW 2 major objections 5 minor 19 references

Mull it over: mutation testing based on LLVM

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Mull is an open-source mutation testing tool that works on LLVM IR instead of source code, letting one tool target any language that compiles to LLVM while recompiling only the mutated bitcode for speed.

desk verdict A solid, honest tool paper about an LLVM-IR mutation tester that overclaims Swift support in the abstract; the rest of the paper is credible and worth an actual referee. read the letter →

arxiv 1908.01540 v1 pith:BTTTTFQW submitted 2019-08-05 cs.SE

Alex Denisov , Stanislav Pankevich This is my paper · ORCID

classification cs.SE

keywords mutationtestingLLVMintermediaterepresentationjust-in-timecompilationdynamiccalltreedistancelanguageindependencefault-based

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents Mull, an open-source mutation testing tool that modifies LLVM intermediate representation (bitcode) rather than source code, and uses LLVM's JIT engine to compile and run the results. The authors' central claim is that this design gives two capabilities source-level tools lack: language independence, since any language that compiles to LLVM IR can be analysed with the same mutation operators, and fine-grained control, since only the mutated fragment of bitcode is recompiled for each mutant. They also use runtime instrumentation to build a dynamic call tree, so mutations can be filtered by distance from the test that reaches them, cutting the number of mutants that need to be run. The paper reports applying Mull to RODOS, OpenSSL, and LLVM, with hot-cache runs on a large LLVM test suite taking under two hours. If this holds, Mull offers a practical route to mutation testing for compiled languages where mature tools were previously scarce.

What carries the argument

The central mechanism is LLVM IR as both the mutation surface and the unit of compilation. Mull instruments each function, JIT-compiles the instrumented bitcode, runs tests to build a dynamic call tree, and for each mutation point recompiles only the mutated bitcode file and links it with previously compiled objects. This gives it language independence, because mutators are defined on LLVM instructions rather than syntax, and a speed advantage through partial recompilation, while mutation distance derived from the call tree prunes the mutant set.

What would settle it

A direct comparison on the same LLVM test suite, timing Mull's hot-cache run against a source-level mutation testing tool that uses incremental compilation, would settle whether IR-level partial recompilation is actually faster than the alternative.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that mutation testing can be carried out at the LLVM IR level with JIT execution, and that this combination is both language-agnostic and faster per mutant than full recompilation. Mutations are applied to LLVM instructions (for example, add to sub, condition negation, or removal of a void call), and only the bitcode file containing a mutation is recompiled and linked against already compiled objects in memory. To avoid running unreachable mutants, Mull instruments every function, runs each test, and builds a dynamic call tree; mutation distance then lets users ignore mutations too far from a test. The evaluation on RODOS, OpenSSL, and LLVM is presented as evidence of practical applicability, with dry-run timing estimates and on-disk caching offered as ways to manage the cost. The tool's own scope is bounded by LLVM JIT limitations, notably thread-local storage and Objective-C runtime support.

Load-bearing premise

Everything rests on LLVM JIT being able to compile and execute a project's instrumented and mutated bitcode; projects that use thread-local storage or the Objective-C runtime cannot currently run under Mull, so the advertised language independence is conditional on that engine support.

Editorial extensions

If this is right

A single set of mutation operators can serve C, C++, Rust, Swift, and other LLVM-targeting languages; adding a language requires only a test-framework adapter.
Because only mutated bitcode is recompiled and linked in memory, iterative mutation-testing sessions on large projects become feasible, with on-disk caching making repeat runs faster.
The dynamic call tree gives users a tunable mutation distance, allowing a project to trade thoroughness against runtime.
The SQLite output and HTML reporting decouple analysis from presentation, so Mull can be embedded in continuous integration or developer tooling.
Mutation testing for compiled languages moves from a research exercise to practical tooling for projects that can avoid the documented JIT limitations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the LLVM IR approach matures, mutation operators could be shared across all LLVM languages automatically, so a mutation operator added for C also applies to Rust and Swift without new tooling.
The documented junk and stray mutations suggest that IR-level tools still need source-line filtering or richer pattern matching; a hybrid that maps IR instructions back to source constructs could make results more meaningful.
Extending Mull to avoid JIT limitations, for example by falling back to ahead-of-time object compilation for projects using thread-local storage or the Objective-C runtime, would directly test how much of the performance claim survives outside the current sandbox.
Because Mull's per-test isolation and plugin architecture fit the needs of automated test generation, it could combine with search-based techniques to find tests that kill surviving mutants.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What you should know: this is a real tool paper, not vaporware. Mull is open source, it builds on LLVM IR and JIT in a genuinely different way from the usual source/AST mutation testers, and the authors ran it against RODOS, OpenSSL, and LLVM. The abstract oversells Swift support, but the body is candid about that and about other limitations. If you are in the software testing world, this is worth your time.

What is actually new: the architecture. Mutating LLVM IR instead of source, using JIT to avoid disk I/O, recompiling only the modified bitcode file, building a dynamic call tree to prune unreachable mutants, and adding a mutation distance parameter. The plugin design for test frameworks is pragmatic and lets them handle OpenSSL's custom test harness without special-casing it in the core. The evaluation is real data on real projects, and the dry-run mode plus caching are sensible engineering. Acknowledging junk and stray mutations, and the TLS/Objective-C JIT limits, is more honest than most tool papers get.

The soft spots are in proportion. First, the abstract says Mull can work with C, C++, Rust, or Swift; Section VI.B says Mull does not yet fully support Objective-C and Swift because LLVM JIT lacks the Objective-C runtime. That is a real overstatement, but it is a wording problem, not a fatal design flaw. The whole point of working on LLVM IR is that the design is language-agnostic; the missing support is an implementation maturity issue. Still, they should fix the abstract and either present the CryptoSwift and rustc-demangle data hinted at in the conclusion or drop those examples from the claims. Second, there is no performance comparison against muCpp or any other C/C++ mutation tool. The 'faster processing' claim is asserted, not benchmarked. They explicitly defer this to future work, which is honest, but for a tool paper it is the difference between conditional and strong acceptance. Third, the mutation operators are standard; the novelty is the implementation, not the mutation theory. That is fine, but it should be stated more plainly.

Who is this for? Researchers and practitioners in mutation testing, and people building compiler-based analysis tools. A serious referee should engage with it. I would want the abstract reworded and a comparison with at least one existing tool before publication, but even without that the paper is a useful, citable description of a working system.

Referee Report

2 major / 5 minor

Summary. The paper describes Mull, an open-source mutation testing tool built on LLVM IR and JIT compilation. The core idea is to perform mutations at the IR level, use dynamic instrumentation to build a dynamic call tree for coverage-guided mutant selection, and reuse compiled object files with partial recompilation of mutated bitcode. The tool supports pluggable test framework adapters and a set of basic mutation operators. The evaluation applies Mull to test suites from RODOS, OpenSSL, and LLVM, reporting numbers of mutants, test runs, and wall-clock times for cold and hot cache runs. The paper claims two distinctive capabilities: language independence across LLVM front ends, and fine-grained control leading to faster processing of mutated programs.

Significance. Mull appears to be a working, open-source mutation testing tool with a clean architecture; if the claims are substantiated, it could be a useful contribution for C/C++ developers and, with adapters, for other LLVM-based languages. The paper provides enough implementation detail to be replicable, and the availability of source code is a clear strength. The evaluation on real-world projects such as OpenSSL and LLVM is also a positive feature. However, the two headline claims are currently stronger than the evidence: language independence is undercut by the stated lack of Swift/Objective-C support and by evaluations restricted to C/C++, and the performance advantage is asserted without a comparative baseline. These need qualification or additional evidence before the paper's central claims can be accepted.

major comments (2)

[Abstract vs. Section VI.B] The abstract states: "Mull can work with code written in any programming language that supports compilation to LLVM IR, such as C, C++, Rust, or Swift." In contrast, Section VI.B says: "The latter limitation is the only reason why Mull does not yet fully support Objective-C and Swift programming languages." This is a direct internal contradiction. Because language independence is one of the two named capabilities of Mull, the paper must reconcile these claims. Possible fixes are to weaken the abstract to say "in principle" or "with language-specific adapters," to describe what partial Swift support exists, or to include evaluation data for at least one non-C/C++ LLVM-based language. Without such a change, the language-independence claim as written is unsupported by the paper's own evidence.
[Abstract and Section III.B] The abstract claims that recompiling only modified IR fragments "results in faster processing of mutated programs," and Section III.B asserts that partial recompilation "helps to increase performance." However, no comparison is made against an alternative: there is no baseline from a whole-program recompilation variant of Mull, nor from existing mutation testing tools such as the C++ tools cited in the paper. Tables II, III, and IV report absolute execution times only. The speedup claim is therefore not empirically demonstrated. The authors should either provide a controlled comparison (for example, a mode that recompiles the whole bitcode file rather than the mutated fragment) or explicitly restrict the claim to a qualitative design advantage.

minor comments (5)

[Table III] The row labeled "test test" appears to be a typo; it should likely be "test_test" or another OpenSSL test-suite name. Please verify and correct the name for clarity.
[Table IV] The rows labeled "All Tests" appear twice with different mutation distances; consider labeling the rows with their distance setting (for example, "All Tests (distance = 25)" and "All Tests (distance = 2)") to avoid confusion.
[Section VI.A] The terms "junk mutation" and "stray mutation" are introduced with a reference to a tweet; consider giving a more formal definition and a more accessible reference, as these concepts are important for the limitations discussion.
[Conclusion] The list of analyzed projects in the Conclusion includes CryptoSwift and rustc-demangle, but no results from those projects are reported in Section V. If these were only preliminary tests, please say so explicitly; otherwise, provide at least a brief summary of the findings.
[Abstract] The novelty claim "To our knowledge, no existing mutation testing tool provides these capabilities for compiled programming languages" is a strong assertion that is not backed by a comprehensive survey. Suggest softening the wording to avoid overclaiming, since only Pitest and a survey are cited.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper is an empirical tool report with no derivation chain that reduces to its own inputs.

full rationale

This manuscript reports on the design and evaluation of the Mull mutation-testing tool. There is no formalism, fitted model, or derivation that could be circular. The two headline capabilities (language independence and fine-grained control via LLVM IR/JIT) are architectural claims, and the paper supports them with concrete measurements on RODOS, OpenSSL, and LLVM, plus admitted limitations in Section VI. The mention of self-analysis (Mull on Mull) appears only as one item in a list of projects and is not used to justify any claim. External grounding is cited (Pitest, Jia-Harman survey, LLVM documentation), and the JIT limitations are traced to LLVM bug reports rather than to the authors' own prior work. The abstract's mention of Swift alongside the Section VI.B statement that Mull 'does not yet fully support Objective-C and Swift' is an internal consistency problem for the language-independence claim, but it is a correctness/evidence concern, not a circularity: no prediction is fitted from data, no parameter is renamed as a result, and no self-citation is load-bearing. The paper is self-contained against external benchmarks, so the appropriate circularity score is 0.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central claim is a software engineering claim about a tool design. No quantities are fitted to data, and the tool introduces no new natural entities. The evaluation depends on configuration choices and platform-specific toolchain behavior, which are listed as axioms and free parameters above.

free parameters (1)

Mutation distance threshold = varies from 2 to 25 depending on test suite
In Section V, each test suite is configured with a hand-chosen mutation distance (Dist. column) that limits how far from the test function a mutation can be. This choice changes the mutant count and total runtime, but it is an experimental setting, not a fitted model parameter.

assumptions (4)

domain assumption LLVM IR is a sufficient abstraction for implementing mutation operators.
The paper defines mutation operators over LLVM IR instructions (Section IV), assuming each IR-level change corresponds to a source-level fault. Section VI.A admits this is imperfect, introducing 'junk mutations' that have no source-level equivalent.
domain assumption The LLVM JIT engine can compile and execute the tested program and its mutations in-memory.
Section III.B relies on JIT for all execution, and Section VI.B lists unsupported features (TLS, Objective-C runtime), so this assumption is only conditionally true.
domain assumption The dynamic call tree from instrumentation correctly captures which code is reachable from each test.
Section III.A uses the dynamic call tree to decide which mutants to create and run; if reachability information is incomplete, mutants may be skipped.
domain assumption Fork-based process isolation reliably distinguishes killed, survived, crashed, and timed-out mutants.
Section III.C describes exit-code 227 and alarm-based timeout handling; this relies on the OS and the child process behaving as specified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Mull it over: mutation testing based on LLVM." pith.science (2026). https://pith.science/paper/BTTTTFQW

@misc{pith2026190801540,
  author       = {Pith},
  title        = {Pith review of: Mull it over: mutation testing based on LLVM},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BTTTTFQW}},
  note         = {Machine review of arXiv:1908.01540}
}

read the original abstract

This paper describes Mull, an open-source tool for mutation testing based on the LLVM framework. Mull works with LLVM IR, a low-level intermediate representation, to perform mutations, and uses LLVM JIT for just-in-time compilation. This design choice enables the following two capabilities of Mull: language independence and fine-grained control over compilation and execution of a tested program and its mutations. Mull can work with code written in any programming language that supports compilation to LLVM IR, such as C, C++, Rust, or Swift. Direct manipulation of LLVM IR allows Mull to do less work to generate mutations: only modified fragments of IR code are recompiled, and this results in faster processing of mutated programs. To our knowledge, no existing mutation testing tool provides these capabilities for compiled programming languages. We describe the algorithm and implementation details of Mull, highlight current limitations of Mull, and present the results of our evaluation of Mull on real-world projects such as RODOS, OpenSSL, LLVM.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 18 canonical work pages

[1]

Github topics: Mutation testing

“Github topics: Mutation testing.” [Online]. Availabl e: https://github.com/topics/mutation-testing

work page
[2]

Coles, “Pitest.” [Online]

H. Coles, “Pitest.” [Online]. Available: http://pites t.org

work page
[3]

Schirp, “Mutant.” [Online]

M. Schirp, “Mutant.” [Online]. Available: https://git hub.com/mbj/mutant

work page
[4]

Llvm: A compilation framework fo r lifelong program analysis & transformation,

C. Lattner and V . Adve, “Llvm: A compilation framework fo r lifelong program analysis & transformation,” in Proceedings of the International Symposium on Code Generation and Optimization: Feedback-d irected and Runtime Optimization , ser. CGO ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 75–. [Online]. Availa ble: http://dl.acm.org/citation....

arXiv 2004
[5]

Denisov and S

A. Denisov and S. Pankevich, “Mull.” [Online]. Availabl e: https://github.com/mull-project/mull

work page
[6]

Apache license,

“Apache license,” Apache Software Foundation. [Online ]. Available: https://www.apache.org/licenses/LICENSE-2.0

work page
[7]

LL VM Language Reference Manual: In- struction Reference

“LL VM Language Reference Manual: In- struction Reference.” [Online]. Available: https://releases.llvm.org/3.9.0/docs/LangRef.html#instruction-reference

work page
[8]

Pitest: Available mutation operations

H. Coles, “Pitest: Available mutation operations.” [On line]. Available: http://pitest.org/quickstart/mutators/

work page

Show all 19 references

[9]

Available: https://en.wikipedia.org/wiki/Rodos (operating system)

“RODOS.” [Online]. Available: https://en.wikipedia.org/wiki/Rodos (operating system)
[10]

OpenSSL

“OpenSSL.” [Online]. Available: https://www.openss l.org
[11]

Available: https://llvm.org

“LL VM.” [Online]. Available: https://llvm.org
[12]

CppUnit

“CppUnit.” [Online]. Available: https://sourceforg e.net/projects/cppunit/
[13]

LL VM Link Time Optimization: Design and Implementati on

“LL VM Link Time Optimization: Design and Implementati on.” [Online]. Available: https://llvm.org/docs/LinkTimeOptimization.html
[14]

GoogleTest

“GoogleTest.” [Online]. Available: https://github. com/google/googletest
[15]

MCJIT TLS support: Cannot select: X86ISD::WrapperRI P

“MCJIT TLS support: Cannot select: X86ISD::WrapperRI P.” [Online]. Available: https://bugs.llvm.org/show bug.cgi?id=21431
[16]

Junk Mutations

H. Coles, “Junk Mutations.” [Online]. Available: https://twitter.com/0hjc/status/478896988784963584
[17]

[llvm-dev] Is it possible to execute Objective- C code via LL VM JIT?

“[llvm-dev] Is it possible to execute Objective- C code via LL VM JIT?” [Online]. Available: http://lists.llvm.org/pipermail/llvm-dev/2016-October/106218.html

2016
[18]

Assessment of class mut ation operators for c++ with the mucpp mutation system,

P . Delgado-Prez, I. Medina-Bulo, F. Palomo-Lozano, A. Garca- Domnguez, and J. Domnguez-Jimnez, “Assessment of class mut ation operators for c++ with the mucpp mutation system,” vol. 81, p . 169184, 01 2017

2017
[19]

An analysis and survey of the devel opment of mutation testing,

Y . Jia and M. Harman, “An analysis and survey of the devel opment of mutation testing,” IEEE Transactions on Software Engineering , vol. 37, no. 5, pp. 649–678, Sept 2011

2011

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

Github topics: Mutation testing

“Github topics: Mutation testing.” [Online]. Availabl e: https://github.com/topics/mutation-testing

work page

[2] [2]

Coles, “Pitest.” [Online]

H. Coles, “Pitest.” [Online]. Available: http://pites t.org

work page

[3] [3]

Schirp, “Mutant.” [Online]

M. Schirp, “Mutant.” [Online]. Available: https://git hub.com/mbj/mutant

work page

[4] [4]

Llvm: A compilation framework fo r lifelong program analysis & transformation,

C. Lattner and V . Adve, “Llvm: A compilation framework fo r lifelong program analysis & transformation,” in Proceedings of the International Symposium on Code Generation and Optimization: Feedback-d irected and Runtime Optimization , ser. CGO ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 75–. [Online]. Availa ble: http://dl.acm.org/citation....

arXiv 2004

[5] [5]

Denisov and S

A. Denisov and S. Pankevich, “Mull.” [Online]. Availabl e: https://github.com/mull-project/mull

work page

[6] [6]

Apache license,

“Apache license,” Apache Software Foundation. [Online ]. Available: https://www.apache.org/licenses/LICENSE-2.0

work page

[7] [7]

LL VM Language Reference Manual: In- struction Reference

“LL VM Language Reference Manual: In- struction Reference.” [Online]. Available: https://releases.llvm.org/3.9.0/docs/LangRef.html#instruction-reference

work page

[8] [8]

Pitest: Available mutation operations

H. Coles, “Pitest: Available mutation operations.” [On line]. Available: http://pitest.org/quickstart/mutators/

work page