Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Leveraging Propagated Infection to Crossfire Mutants

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A small, crossfire-optimized set of added assertions can kill every surviving mutant that existing test data can detect.

desk verdict A novel, well-scoped test-amplification paper whose central 'we can kill' claim is inferred from memory-state differences, not from actually running the augmented tests—so the headline numbers are unvalidated upper bounds. read the letter →

arxiv 2411.09846 v1 pith:DRCESBP5 submitted 2024-11-14 cs.SE

Hang Du , Vijay Krishna Palepu , James A. Jones This is my paper · ORCID
classification cs.SE
keywords mutationtestingassertionamplificationsurvivingmutantscrossfirememory-stateanalysisobjectgraphdiffingtestoraclegenerationRIPRmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that many surviving mutants—up to 84% in its study—are not hard to kill because they need new test inputs; they survive because existing tests lack assertions that check values the mutation already corrupted. It shows that a fine-grained comparison of memory states between the original program and each mutant can pinpoint exactly where the corrupted value is reachable from a test's variables. From those locations it builds a mutant-to-location matrix and selects assertions greedily, so that one assertion kills many mutants at once (crossfire). Across ten projects, 1,684 assertions—1.1% of the 147,989 candidates—suffice to kill all 5,504 surviving mutants the analysis judged detectable, with a 6.1x average gain in killed mutants per amplified test.

What carries the argument

The carrier of the argument is the A-model, which splits mutation work into analysis (how existing tests kill mutants) and testing (how new assertions kill survivors), with crossfire opportunities at test, variable, and assertion granularities. The mechanism that makes it concrete is fine-grained memory-state analysis: the original program is run N times to label each reachable object-graph node deterministic or flaky, each mutant is run once, and a breadth-first diff of the matched graphs records every node whose value differs from the original. These differences are collected into a mutant-to-memory-location matrix; an assertion on one location is a candidate for every mutant that differs there. Selection then applies greedy set-cover strategies—most-capable assertion, most-capable variable, most-capable test—with a shortest-access-path filter that avoids brittle deep assertions.

What would settle it

A reader could take one subject, such as spotify-web-api, and re-run the 187 assertions from the assertion-greedy strategy against the 243 predicted-killable mutants. If fewer than all 243 mutated runs fail, the memory-state-difference criterion overestimates how many surviving mutants assertions can kill; if all fail, the crossfire claim is confirmed.

Watch

Extended reading notes

Core claim

The central discovery is that the revealability condition of the RIPR model is often the only missing link for surviving mutants. When a mutant runs under an existing test, the infection frequently propagates back to objects the test can see, but no assertion checks those objects. The paper's memory-state analysis runs each surviving mutant and the original program under the same test, serializes all variables reachable from the test scope, and diffs the object graphs after filtering nondeterministic locations. Every differing node is an assertion candidate: assert the original value there and the mutant dies. Because one corrupted node can be shared by several mutants (they all alter the same field in different ways), the technique then treats assertion selection as a set-cover problem and greedily picks assertions, variables, or tests that cover the most mutants—producing a 'crossfire' where a single added assertion kills several survivors. The paper reports that all predicted-detectable survivors can be killed this way with 1.1% of the candidate assertions, concentrated in fewer tests.

Load-bearing premise

The load-bearing premise is that a memory-state difference found by comparing serialized object graphs of the original and mutant programs is enough to guarantee that adding an assertion of the original value at that location will kill the mutant when the amplified test is re-run.

Editorial extensions

If this is right

  • Mutation-testing reports can change from 'here are surviving mutants' to 'here is the assertion to add to this test to kill them.'
  • The review burden of strengthening a test suite drops to 1,684 hand-checkable assertions across 900 tests instead of 147,989 candidates.
  • Because up to 84% of survivors are killable with existing test data, many surviving mutants should be treated as oracle gaps rather than as equivalent mutants or infeasible cases.
  • The test-greedy strategy concentrates amplification within fewer tests (for example, 23 tests for spotify-web-api instead of 48.6) with almost no extra assertions.
  • The 6.1x crossfire factor means each incremental developer action buys more mutation score than non-crossfire amplification does.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the killability criterion transfers, the same matrix could also prioritize mutants: ones with shallow, first-party infections are more likely to be worth a developer's attention than ones requiring deep or third-party access paths.
  • The 1.1% figure is tied to the greedy objective of minimizing assertions and tests; a developer who instead wants to minimize runtime overhead or flaky-test risk might select a different subset among the 147,989 candidates and trade more assertions for safer tests.
  • The technique could be applied to type systems or property-based testing: an automatically inferred invariant from the original state could act as the oracle, turning 'expected value' assertions into general predicates.
  • A testable extension is to run the amplified tests in continuous integration over many builds and measure whether the added assertions cause failures on future real faults, not just on mutants.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes an approach to strengthen test suites against surviving mutants by augmenting existing tests with assertions derived from fine-grained memory-state analysis. The authors instrument original and mutated programs, record object graphs of memory states reachable from test variables, classify locations as deterministic via N=10 repeated original runs, and treat any deterministic difference between original and mutant state as an infection location that yields an assertion candidate. They introduce an 'A-model' to characterize crossfire effects at the levels of tests, variables, and assertions, and present three greedy strategies that select a small set of assertion candidates to kill all 'killable' surviving mutants. The empirical evaluation on 10 open-source Java projects reports that 19-84% of surviving mutants are killable, that all 5,504 such mutants can be killed with 1,684 of 147,989 assertion candidates, and that the selected assertions provide a crossfire factor of 6.1x over non-crossfire approaches.

Significance. If the findings hold, the paper would make a substantial contribution to mutation testing and test amplification: it would demonstrate that many surviving mutants can be killed without new test data, purely by adding assertions to existing tests, and that crossfire-aware selection can concentrate the required changes in a small number of tests and variables. The work is also valuable as a large empirical study: it covers 46,958 mutants and over 1.2 million mutated test runs, and the authors ship an implementation and dataset to support reproducibility. The A-model is a useful conceptual framework for reasoning about crossfire at multiple granularities. The main concern is that the central empirical claim is not validated by executing the augmented tests against the mutants; the reported 'kill' counts are predictions inferred from state differences.

major comments (3)
  1. The central claim that the technique 'can kill all surviving mutants that are detectable with existing test data' is not yet substantiated by actual test executions. In the approach, Step 7 filters mutant state to deterministic locations, Step 8 labels any deterministic difference from the original as an infection location, and Step 10 records the mutants that a candidate assertion 'would kill.' The evaluation in Section V-B describes state collection and strategy application but never states that the generated assertions were inserted into the tests and re-run against the mutants. Consequently, the #Killable counts in Table II and the 'kill all 5,504 surviving, killable mutants' claim in RQ4 (Table III) are counts of predicted kills based on state-difference inference, not measured kills. This is load-bearing because a serialized-object-graph difference at test completion does not guarantee that an assertion checking the original expected value at that location will fail on the mutant in a real re-run: the graph is collected with XStream and compared by breadth-first traversal, so object identity, collection ordering, and access-path method side effects may invalidate the inference. Please run the augmented tests against the mutants and report measured kills, or revise the abstract and conclusions to state that the results are predictions from state-diff analysis.
  2. The choice of N=10 repeated runs to classify memory locations as deterministic is not justified, and no sensitivity analysis is provided. This parameter is load-bearing because Step 7 removes all nondeterministic locations before comparing original and mutant states; if a location is actually nondeterministic but happens to agree across 10 original runs, it will be misclassified as deterministic and may be counted as an infection location, inflating #Killable. Conversely, a genuinely deterministic location could be misclassified as nondeterministic due to a rare flaky value. The manuscript reports in Section V-B that 'we found no test flakiness in the subjects' test suites,' but N=10 is not strong evidence of absence of flakiness. Please provide a sensitivity analysis over N (e.g., 5, 20, 50) or otherwise justify the choice.
  3. The evaluation of the three crossfire strategies compares only against their own non-crossfire counterpart and reports the crossfire factor as the average number of mutants killed per selected element relative to the non-crossfire approach. The paper does not compare with existing assertion-amplification techniques, such as DSpot, which is cited in the related work. Since the central contribution is positioned as a technique for assertion amplification, an external baseline is needed to assess whether the reported gains are due to crossfire optimization or simply to the ability to detect state differences. If the authors intend the contribution to be only the crossfire-selection mechanism, they should state that boundary explicitly; as written, the abstract claims a general amplification improvement of 6x.
minor comments (4)
  1. The abstract's '1.1%' figure is not explicitly derived in the results section; the ratio 1,684/147,989 is given in the RQ4 text, but the percentage should be stated there and tied to the abstract.
  2. The color legend in Figure 4(g) is difficult to read at the scale used in the small multiples; please enlarge the legend or add numeric labels to the color scale.
  3. The 'Ways of killing each killable mutant' columns present averages without measures of dispersion; reporting standard deviations or distributions would strengthen the claim that the reported averages are representative.
  4. The internal-validity paragraph discusses duplicate mutants and execution nondeterminism but does not address the risk that the N=10 determinism filter may misclassify nondeterministic locations; a sentence acknowledging this limitation and pointing to the sensitivity analysis would be appropriate.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline 'kill' result is a set-cover property of the infection matrix, not an executed-test result.

  1. self definitional [Section IV-A Steps 8-10; Section IV-B; Section VI RQ2 and Table II]
    "8) Compare deterministic state of the original program SP,d with the deterministic state of the mutant SMi,d to identify all locations SMi,i that reveal infection. ... 10) Performing strategies on M to optimize crossfiring to produce a list of assertion candidates, and for each assertion candidate the list of surviving mutants that it would kill. ... a count of surviving mutants (#Surviving) alongside those that are killable (#Killable), as identified by our analysis."

    The 'killable' population is defined as the mutants for which Steps 7-8 found at least one deterministic state difference SMi,i. Step 10 turns each such difference into an assertion candidate 'that it would kill,' and the greedy strategies in Section IV-B iterate 'until all assertion-actionable surviving mutants are killed.' Thus the claim that the selected assertions kill all detectable surviving mutants is, by construction, the statement that the greedy set-cover loop covers every mutant row in the infection matrix. The paper never reports executing the augmented tests against the mutants: Section V-B describes recording memory states, comparing object graphs, and running selection strategies, but no step inserts the generated assertions and reruns the tests to observe actual failures.

full rationale

The core circular step is the equation of 'state difference found by the paper's own memory-state analysis' with 'killable by adding an assertion,' followed by reporting the greedy selection's coverage of that same set as an empirically measured kill rate. Because the augmented tests are not executed, the 19%-84% killable counts and the 'kill all killable' results are not independent measurements; they are properties of the constructed infection matrix. This is partial circularity: the state-difference data and the set-cover optimization have genuine empirical content, and the paper is transparent that the candidates are those that 'would kill' or are 'assertion-actionable.' However, the headline abstract claim uses the word 'kill' as if the amplification had been validated by execution. I found no load-bearing self-citation circularity: the RIPR model is cited to Li and Offutt, crossfire to Smith and Williams, and the authors' own prior work is used for context and motivation rather than as a uniqueness theorem forcing the present result. The score is 6 rather than 8 because the optimization itself (selecting 1,684 of 147,989 candidates) is a nontrivial combinatorial result; what reduces by construction is the claimed killing outcome, not the entire technique.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The central result depends on the object-graph diff as a proxy for test failure and on the N=10 threshold for nondeterminism. No numerical parameters are fitted to data, but the N=10 choice is a hand-selected experimental threshold. The technique introduces no new physical or conceptual entities beyond the descriptive A-model.

free parameters (1)
  • N (number of repeated non-mutated test runs used to classify deterministic vs nondeterministic memory locations) = 10
    Adopted in Section IV-A Step 4 without sensitivity analysis; if some locations vary across more than 10 runs, they would be mislabeled deterministic and could produce flaky assertion candidates.
assumptions (4)
  • domain assumption The RIPR model (reachability, infection, propagation, revealability) is an adequate framework for mutant detection.
    Used in Sections I and II to justify why assertion amplification can turn an executing-but-passing test into a mutant-killing test.
  • domain assumption PIT's DEFAULTS mutation operators pre-filter equivalent and duplicate mutants.
    The paper relies on this in Section V-B to ensure that surviving mutants are not artifacts of equivalent mutants, citing PIT's mutator documentation [24].
  • domain assumption Serialized object graphs and breadth-first comparison faithfully capture all relevant memory-state differences reachable from test variables.
    This is the core measurement in Section IV-A; if XStream configurations omit relevant heap state or if accessor side effects alter state, the candidate assertions would be unreliable.
  • domain assumption Adding an assertion of the original expected value at a detected infected location will fail on the mutant and pass on the original.
    Central inference in Section IV-A Step 8 and RQ2; it is not validated by executing the augmented tests in Section V-B, so practical failures (flaky accessors, side effects, or unstable locations) remain possible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Leveraging Propagated Infection to Crossfire Mutants." pith.science (2026). https://pith.science/paper/DRCESBP5

@misc{pith2026241109846,
  author       = {Pith},
  title        = {Pith review of: Leveraging Propagated Infection to Crossfire Mutants},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DRCESBP5}},
  note         = {Machine review of arXiv:2411.09846}
}
read the original abstract

Mutation testing was proposed to identify weaknesses in test suites by repeatedly generating artificially faulty versions of the software (mutants) and determining if the test suite is sufficient to detect them (kill them). When the tests are insufficient, each surviving mutant provides an opportunity to improve the test suite. We conducted a study and found that many such surviving mutants (up to 84% for the subjects of our study) are detectable by simply augmenting existing tests with additional assertions, or assertion amplification. Moreover, we find that many of these mutants are detectable by multiple existing tests, giving developers options for how to detect them. To help with these challenges, we created a technique that performs memory-state analysis to identify candidate assertions that developers can use to detect the surviving mutants. Additionally, we build upon prior research that identifies ``crossfiring'' opportunities -- tests that coincidentally kill multiple mutants. To this end, we developed a theoretical model that describes the varying granularities that crossfiring can occur in the existing test suite, which provide opportunities and options for how to kill surviving mutants. We operationalize this model to an accompanying technique that optimizes the assertion amplification of the existing tests to crossfire multiple mutants with fewer added assertions, optionally concentrated within fewer tests. Our experiments show that we can kill all surviving mutants that are detectable with existing test data with only 1.1% of the identified assertion candidates, and increasing by a factor of 6x, on average, the number of killed mutants from amplified tests, over tests that do not crossfire.

Figures

Figures reproduced from arXiv: 2411.09846 by the authors.

Figure 1
Figure 1. Process to kill surviving mutants by manual developer effort (top) that does not optimize for crossfiring due to its [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A-model: A Breakdown of Mutant Crossfire Effects [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Mutant Killing Strategies switch contexts between different amplification locations in the test code. As such, we can minimize the number of assertions (e.g., checks for f2 and f8), minimize the number of variable checks (e.g., checks for var 1) and minimize the number of tests to perform such assertion amplification (e.g., improve Test 1 only rather than both tests) while achieving the same mutation score. As such,… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Test and Assertion Crossfiring Capabilities [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Assertion Examples 147, 989 to 1, 684 among 900 tests, while still being able to kill all 5, 504 surviving, killable mutants across the ten subject programs. This provides a crossfire factor of 6.1 (i.e., 900 tests crossfiring 5, 504 surviving mutants). Representative …

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PRIMG : Efficient LLM-driven Test Generation Using Mutant Prioritization

    cs.SE 2025-05 reject novelty 4.0 of 10

    PRIMG combines mutant prioritization with LLM-based iterative test refinement, and its evaluation on three Solidity projects suggests prioritized mutant targeting yields more killed mutants than random targeting.

Reference graph

Works this paper leans on

53 extracted references · 41 canonical work pages · cited by 1 Pith paper

  1. [1]

    Mutation-driven generation of unit tests and oracles,

    G. Fraser and A. Zeller, “Mutation-driven generation of unit tests and oracles,” in Proceedings of the 19th international symposium on Software testing and analysis , 2010, pp. 147–158

    2010

  2. [2]

    Whole test suite generation,

    G. Fraser and A. Arcuri, “Whole test suite generation,” IEEE Transac- tions on Software Engineering , vol. 39, no. 2, pp. 276–291, 2013

    2013

  3. [3]

    Dspot: Test amplification for automatic assessment of computational diversity,

    B. Baudry, S. Allier, M. Rodriguez-Cancio, and M. Monperrus, “Dspot: Test amplification for automatic assessment of computational diversity,” 03 2015

    work page 2015

  4. [4]

    Automatic test improvement with dspot: a study with ten mature open-source projects,

    B. Danglot, O. L. Vera-P ´erez, B. Baudry, and M. Monperrus, “Automatic test improvement with dspot: a study with ten mature open-source projects,” Empirical Software Engineering , vol. 24, pp. 2603–2635, 2019

    work page 2019

  5. [5]

    Test data generation techniques for mutation testing: A systematic mapping,

    F. C. Souza, M. Souza, M. Papadakis, V . Durelli, and M. Delamaro, “Test data generation techniques for mutation testing: A systematic mapping,” 04 2014

    work page 2014

  6. [6]

    The oracle problem in software testing: A survey,

    E. T. Barr, M. Harman, P. McMinn, M. Shahbaz, and S. Yoo, “The oracle problem in software testing: A survey,” IEEE Transactions on Software Engineering, vol. 41, no. 5, pp. 507–525, 2015

    2015

  7. [7]

    A snowballing literature study on test amplification,

    B. Danglot, O. Vera-Perez, Z. Yu, A. Zaidman, M. Monperrus, and B. Baudry, “A snowballing literature study on test amplification,” Journal of Systems and Software , vol. 157, p. 110398, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0164121219301736

    work page 2019

  8. [8]

    An orchestrated survey of methodologies for automated software test case generation,

    S. Anand, E. K. Burke, T. Y . Chen, J. Clark, M. B. Cohen, W. Grieskamp, M. Harman, M. J. Harrold, and P. Mcminn, “An orchestrated survey of methodologies for automated software test case generation,” J. Syst. Softw. , vol. 86, no. 8, pp. 1978–2001, aug 2013. [Online]. Available: https://doi.org/10.1016/j.jss.2013.02.061

    doi:10.1016/j.jss.2013.02.061 1978

Show all 53 references
  1. [9]

    Evosuite: automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT Symposium and the 13th European Conference on Foundations of Software Engineering , ser. ESEC/FSE ’11. New York, NY , USA: Association for ...

    2011

  2. [10]

    State of mutation testing at google,

    G. Petrovi ´c and M. Ivankovi ´c, “State of mutation testing at google,” in Proceedings of the 40th international conference on software engineer- ing: Software engineering in practice , 2018, pp. 163–171

    2018

  3. [11]

    Please fix this mutant: How do developers resolve mutants surfaced during code review?

    G. Petrovi ´c, M. Ivankovi ´c, G. Fraser, and R. Just, “Please fix this mutant: How do developers resolve mutants surfaced during code review?” in 2023 IEEE/ACM 45th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP) , 2023, pp. 150–161

    2023

  4. [12]

    What it would take to use mutation testing in industry— a study at facebook,

    M. Beller, C.-P. Wong, J. Bader, A. Scott, M. Machalica, S. Chandra, and E. Meijer, “What it would take to use mutation testing in industry— a study at facebook,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Software Engineering in Practice (ICSE- SE...

    2021

  5. [13]

    On guiding the augmentation of an auto- mated test suite via mutation analysis,

    B. H. Smith and L. Williams, “On guiding the augmentation of an auto- mated test suite via mutation analysis,” Empirical software engineering, vol. 14, no. 3, pp. 341–369, 2009

    2009

  6. [14]

    Code defenders: Crowdsourcing effective tests and subtle mutants with a mutation testing game,

    J. M. Rojas, T. D. White, B. S. Clegg, and G. Fraser, “Code defenders: Crowdsourcing effective tests and subtle mutants with a mutation testing game,” in 2017 IEEE/ACM 39th International Conference on Software Engineering (ICSE), 2017, pp. 677–688

    2017

  7. [15]

    Test oracle strategies for model-based testing,

    N. Li and J. Offutt, “Test oracle strategies for model-based testing,” IEEE Transactions on Software Engineering , vol. 43, no. 4, pp. 372– 395, 2017

    2017

  8. [16]

    An empirical analysis of test oracle strategies for model-based testing,

    ——, “An empirical analysis of test oracle strategies for model-based testing,” in 2014 IEEE Seventh International Conference on Software Testing, Verification and Validation, 2014, pp. 363–372

    2014

  9. [17]

    Ripples of a mutation — an empirical study of propagation effects in mutation testing,

    H. Du, V . K. Palepu, and J. A. Jones, “Ripples of a mutation — an empirical study of propagation effects in mutation testing,” in Proceedings of the IEEE/ACM 46th International Conference on Software Engineering , ser. ICSE ’24. New York, NY , USA: Association for Computing M...

    2024

  10. [18]

    An empirical evaluation of the mu- java mutation operators,

    B. H. Smith and L. Williams, “An empirical evaluation of the mu- java mutation operators,” in Testing: Academic and Industrial Con- ference Practice and Research Techniques - MUTATION (TAICPART- MUTATION 2007), 2007, pp. 193–202

    2007

  11. [19]

    Pit: a practical mutation testing tool for java,

    H. Coles, T. Laurent, C. Henard, M. Papadakis, and A. Ventresque, “Pit: a practical mutation testing tool for java,” in Proceedings of the 25th international symposium on software testing and analysis , 2016, pp. 449–452

    2016

  12. [20]

    Mujava: an automated class mutation system,

    Y .-S. Ma, J. Offutt, and Y . R. Kwon, “Mujava: an automated class mutation system,” Software Testing, Verification and Reliability, vol. 15, no. 2, pp. 97–133, 2005

    2005

  13. [21]

    To kill a mutant: An empirical study of mutation testing kills,

    H. Du, V . K. Palepu, and J. A. Jones, “To kill a mutant: An empirical study of mutation testing kills,” in Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis , ser. ISSTA 2023. New York, NY , USA: Association for Computing Machinery, ...

    2023

  14. [22]

    How effective are mutation testing tools? an empirical analysis of java mutation testing tools with manual analysis and real faults,

    M. Kintis, M. Papadakis, A. Papadopoulos, E. Valvis, N. Malevris, and Y . Le Traon, “How effective are mutation testing tools? an empirical analysis of java mutation testing tools with manual analysis and real faults,” Empirical Software Engineering, vol. 23, no. 4, pp. 2426–2...

    2018

  15. [23]

    Mitigating the effects of flaky tests on mutation testing,

    A. Shi, J. Bell, and D. Marinov, “Mitigating the effects of flaky tests on mutation testing,” in Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis , 2019, pp. 112–122

    2019

  16. [24]

    Mutation operators,

    H. Coles, “Mutation operators,” 2023, mutation operators in PIT. [Online]. Available: https://pitest.org/quickstart/mutators/

    2023

  17. [25]

    Asm: a code manipulation tool to implement adaptable systems,

    E. Bruneton, R. Lenglet, and T. Coupaye, “Asm: a code manipulation tool to implement adaptable systems,” Adaptable and extensible component systems , vol. 30, no. 19, 2002. [Online]. Available: http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.117.5769

    2002

  18. [26]

    [Accessed 01-08-2023]

    (2022, December) Xstream - about xstream. [Accessed 01-08-2023]. [Online]. Available: https://x-stream.github.io/

    2022

  19. [27]

    E. R. Tufte, The visual display of quantitative information . Graphics press Cheshire, CT, 2001, vol. 2

    2001

  20. [28]

    Prioritizing mutants to guide mutation testing,

    S. J. Kaufman, R. Featherman, J. Alvin, B. Kurtz, P. Ammann, and R. Just, “Prioritizing mutants to guide mutation testing,” in 2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE), 2022, pp. 1743–1754

    2022

  21. [29]

    Chapter six - mutation testing advances: An analysis and survey,

    M. Papadakis, M. Kintis, J. Zhang, Y . Jia, Y . L. Traon, and M. Harman, “Chapter six - mutation testing advances: An analysis and survey,” ser. Advances in Computers, A. M. Memon, Ed. Elsevier, 2019, vol. 112, pp. 275–378. [Online]. Available: https: //www.sciencedirect.com/s...

    2019

  22. [30]

    Higher order mutation testing,

    Y . Jia and M. Harman, “Higher order mutation testing,” Information and Software Technology, vol. 51, no. 10, pp. 1379–1393, 2009, source Code Analysis and Manipulation, SCAM 2008. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584909000688

    2009

  23. [31]

    Do redundant mutants affect the effectiveness and efficiency of mutation analysis?

    R. Just, G. M. Kapfhammer, and F. Schweiggert, “Do redundant mutants affect the effectiveness and efficiency of mutation analysis?” in 2012 IEEE Fifth International Conference on Software Testing, Verification and Validation, 2012, pp. 720–725

    2012

  24. [32]

    Improving logic-based testing,

    G. Kaminski, P. Ammann, and J. Offutt, “Improving logic-based testing,” Journal of Systems and Software , vol. 86, no. 8, pp. 2002–2012, 2013. [Online]. Available: https://www.sciencedirect.com/ science/article/pii/S0164121212002403 12

    2002

  25. [33]

    Using non-redundant mutation operators and test suite prioritization to achieve efficient and scalable mutation analysis,

    R. Just, G. M. Kapfhammer, and F. Schweiggert, “Using non-redundant mutation operators and test suite prioritization to achieve efficient and scalable mutation analysis,” in 2012 IEEE 23rd International Symposium on Software Reliability Engineering , 2012, pp. 11–20

    2012

  26. [34]

    Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique,

    M. Papadakis, Y . Jia, M. Harman, and Y . Le Traon, “Trivial compiler equivalence: A large scale empirical study of a simple, fast and effective equivalent mutant detection technique,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering , vol. 1, 2015, ...

    2015

  27. [35]

    Evaluating mutation testing alternatives: A collateral experiment,

    M. Kintis, M. Papadakis, and N. Malevris, “Evaluating mutation testing alternatives: A collateral experiment,” in 2010 Asia Pacific Software Engineering Conference. IEEE, 2010, pp. 300–309

    2010

  28. [36]

    Threats to the validity of mutation-based test assessment,

    M. Papadakis, C. Henard, M. Harman, Y . Jia, and Y . Le Traon, “Threats to the validity of mutation-based test assessment,” in Proceedings of the 25th International Symposium on Software Testing and Analysis , ser. ISSTA 2016. New York, NY , USA: Association for Computing Mach...

    2016

  29. [37]

    Selecting fault revealing mutants,

    T. Titcheu Chekam, M. Papadakis, T. F. Bissyand ´e, Y . Le Traon, and K. Sen, “Selecting fault revealing mutants,” Empirical Software Engineering, vol. 25, no. 1, pp. 434–487, 2020

    2020

  30. [38]

    Code defenders: a mutation testing game,

    J. M. Rojas and G. Fraser, “Code defenders: a mutation testing game,” in 2016 IEEE Ninth International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, 2016, pp. 162– 167

    2016

  31. [39]

    Killing stubborn mutants with symbolic execution,

    T. T. Chekam, M. Papadakis, M. Cordy, and Y . L. Traon, “Killing stubborn mutants with symbolic execution,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 30, no. 2, pp. 1–23, 2021. [Online]. Available: https://www.sciencedirect.com/science/ article/pi...

    2021

  32. [40]

    Dynamically discovering likely program invariants to support program evolution,

    M. Ernst, J. Cockrell, W. Griswold, and D. Notkin, “Dynamically discovering likely program invariants to support program evolution,” IEEE Transactions on Software Engineering, vol. 27, no. 2, pp. 99–123, 2001

    2001

  33. [41]

    From daikon to agitator: Lessons and challenges in building a commercial tool for developer testing,

    M. Boshernitsan, R. Doong, and A. Savoia, “From daikon to agitator: Lessons and challenges in building a commercial tool for developer testing,” in Proceedings of the 2006 International Symposium on Software Testing and Analysis , ser. ISSTA ’06. New York, NY , USA: Associatio...

    2006

  34. [42]

    Gassert: A fully automated tool to improve assertion oracles,

    V . Terragni, G. Jahangirova, P. Tonella, and M. Pezz `e, “Gassert: A fully automated tool to improve assertion oracles,” in 2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion), 2021, pp. 85–88

    2021

  35. [43]

    Augmenting automatically generated unit-test suites with regression oracle checking,

    T. Xie, “Augmenting automatically generated unit-test suites with regression oracle checking,” in Proceedings of the 20th European Conference on Object-Oriented Programming, ser. ECOOP’06. Berlin, Heidelberg: Springer-Verlag, 2006, pp. 380–403. [Online]. Available: https://doi...

    2006 doi

  36. [44]

    Dynamic human- in-the-loop assertion generation,

    L. Zamprogno, B. Hall, R. Holmes, and J. M. Atlee, “Dynamic human- in-the-loop assertion generation,” IEEE Transactions on Software Engi- neering, vol. 49, no. 4, pp. 2337–2351, 2023

    2023

  37. [45]

    Automated oracle creation support, or: How i learned to stop worrying about fault propagation and love mutation testing,

    M. Staats, G. Gay, and M. P. Heimdahl, “Automated oracle creation support, or: How i learned to stop worrying about fault propagation and love mutation testing,” in 2012 34th International Conference on Software Engineering (ICSE) . IEEE, 2012, pp. 870–880

    2012

  38. [46]

    Mimicking production behav- ior with generated mocks,

    D. Tiwari, M. Monperrus, and B. Baudry, “Mimicking production behav- ior with generated mocks,” IEEE Transactions on Software Engineering, pp. 1–26, 2024

    2024

  39. [47]

    Mutation-driven generation of unit tests and oracles,

    G. Fraser and A. Zeller, “Mutation-driven generation of unit tests and oracles,” IEEE Transactions on Software Engineering , vol. 38, no. 2, pp. 278–292, 2012

    2012

  40. [48]

    Suggestions on test suite improvements with automatic infection and propagation analysis,

    O. L. Vera-P ´erez, B. Danglot, M. Monperrus, and B. Baudry, “Suggestions on test suite improvements with automatic infection and propagation analysis,” 2019. [Online]. Available: https://arxiv.org/abs/ 1909.04770

    2019 arXiv

  41. [49]

    Assertions are strongly correlated with test suite effectiveness,

    Y . Zhang and A. Mesbah, “Assertions are strongly correlated with test suite effectiveness,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering , 2015, pp. 214–224

    2015

  42. [50]

    Improving oracle quality by detecting brittle assertions and unused inputs in tests,

    C. Huo and J. Clause, “Improving oracle quality by detecting brittle assertions and unused inputs in tests,” in Proceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering, ser. FSE 2014. New York, NY , USA: Association for Computing Mach...

    2014

  43. [51]

    Automatically generating realistic test input from web services,

    M. Bozkurt and M. Harman, “Automatically generating realistic test input from web services,” in Proceedings of 2011 IEEE 6th International Symposium on Service Oriented System (SOSE) , 2011, pp. 13–24

    2011

  44. [52]

    Exploiting common object usage in test case generation,

    G. Fraser and A. Zeller, “Exploiting common object usage in test case generation,” in 2011 Fourth IEEE International Conference on Software Testing, Verification and Validation, 2011, pp. 80–89

    2011

  45. [53]

    Evolutionary improvement of assertion oracles,

    V . Terragni, G. Jahangirova, P. Tonella, and M. Pezz `e, “Evolutionary improvement of assertion oracles,” in Proceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering , 2020, pp. 1178–1189. 13

    2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.