Software Engineering · arXiv:2610.33708 v1 · submitted 24 Oct 2026 · 42 references
Covered Is Not Checked: Which Branch Outcomes Would Your Tests Notice?
The paper says
“Code coverage reports what tests execute, not whether tests detect faults.”
more of the abstract
Checked coverage addresses this gap by tracing code that influences test oracles through dynamic slicing, but slicing is costly, and Java slicers struggle with modern language features and complex programs. We present Veritas, a slicing-free approach that directly tests whether covered behavior is checked. We target object branch coverage (OBC), a compiled-code structural criterion with fault-detection effectiveness comparable to source-level Modified Condition/Decision Coverage (MC/DC), which is required at the highest assurance level of DO-178C. For each covered object-branch outcome, Veritas forces the opposite outcome and credits checking only when the change is executed, a previously passing test fails, and the failure is attributable to that outcome. Thus, Veritas distinguishes execution from demonstrated fault detection without dynamic dependence analysis. We evaluated Veritas on eleven open-source Java projects containing 51,382 object-branch outcomes. We find that 18% of covered outcomes can be changed without failing any test, exposing weaknesses hidden by coverage. Among surviving mutants, about half are never activated by test inputs, while the rest are activated but undetected. Compared with slicing-based checked coverage, Veritas identifies 4,763 cases where tests detect a change that slicing does not credit, while 19% of cases credited by slicing produce no detected failure when changed. Weakening assertions increases this gap and reduces fault detection in every project. Moreover, code locations controlled by weakly checked outcomes are 12.4–54.7 percentage points more likely to have an undetected independent fault than locations controlled by checked outcomes. These findings show that directly testing outcomes provides actionable evidence of oracle effectiveness beyond structural coverage.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. Solid paper that delivers on its promise: a simple, slicing-free way to measure whether covered branch outcomes are actually detected, backed by thorough evaluation on 11 Java projects. This line is wrong
5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 42 references say.
5 minor objections
The paper says, at Section 4.5.2, Table 8
Pith says: minor objection 1 of 5
The caption mentions a 'conservative 95% interval for the difference' but the text does not explain how this interval is computed (e.g., whether it uses a Bonferroni correction or the Wilson-score interval for a difference of proportions). Please clarify the statistical method.
The paper says, at Section 4.2.3
Pith says: minor objection 2 of 5
The qualitative analysis examines 22 sampled activated-but-undetected outcomes (2 per project). The paper correctly notes that sample counts are not population estimates, but the selection strategy (equal per project) may overrepresent small projects and underrepresent large ones. A brief comment on how this might affect the generality of the observed explanations would be helpful.
show 3 more
The paper says, at Section 2.4 / Table 1
Pith says: minor objection 3 of 5
The attribution method (first differing trace entry) fails when traces differ only in length; this is reported as 'ambiguous'. The paper could briefly note how often this case occurs in practice (it appears to be part of the 33 ambiguous outcomes mentioned in Section 4.2.2).
The paper says, at Section 1, Figure 1 caption
Pith says: minor objection 4 of 5
The caption says '(c) The added test ... detects both changes without covering any new branch outcome.' It might be clearer to state explicitly that the added test covers outcome (d2, true) and (d1, false) which were previously uncovered, but the text says it covers no new outcome—please verify and reconcile if needed.
The paper says, at Section 4.4.2, Table 7
Pith says: minor objection 5 of 5
The 'Line coverage' column in Table 5 is defined in the caption but the acronym 'Line coverage' is not defined; it refers to the percentage of surviving mutants for which line-level branch coverage (alone or combined with PIT's verdict for the other mutant) can determine activation. Consider defining this more clearly in the text.
What would settle it
Re-run the RQ4 Experiment 2 analysis after removing all fault sites whose nearest controlling outcome was detected only via a crash (i.e., outcomes checked only because a test threw an unexpected exception). If the difference between gap-controlled and checked-controlled affected-site rates remains positive for all 11 systems (conservative 95% interval above zero in ≥10 of 11), then crash-based detection does not drive the spatial association, strengthening the claim that OBCCG captures oracle-relevant weakness.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Oct 2026
- Reader holds up
- Skeptic agrees
- Editor wrote the verdict above
- Referee minor revision: 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a well-executed paper. The core idea—replace dynamic slicing with direct mutation experiments to measure whether a covered branch outcome is actually detected—is clean and clearly motivated. They implement it as Veritas, evaluate on 11 Java projects (51k+ branch outcomes), and compare with both PIT mutation testing and slicing-based checked coverage (Slicer4J).
What they do well: the metric definitions (OBCC, OBCCG) are operational and avoid free parameters. They handle flaky tests by requiring two identical clean runs. The attribution mechanism (first differing trace entry) is sound for the single-decision-per-mutant design. The comparison with Slicer4J is fair and illuminates the asymmetric disagreement: Veritas finds many more detected outcomes that slicing misses, while 19% of slice-credited outcomes produce no test failure when changed. RQ4 is the key validation: weakening assertions increases OBCCG and decreases held-out fault detection monotonically (Kendall's τ = -1.00 across all 11 projects), and fault sites near activated-but-undetected outcomes are 12–55 percentage points more likely to have a surviving independent fault. The limitations are stated transparently—the proxy assumption, crash counting, association not causation, bounded external validity.
Soft spots, in proportion. The central assumption—that forcing a branch outcome to its opposite is a valid proxy for a real fault—is partially validated with held-out faults, but the qualitative sample shows that about a third of activated-but-undetected outcomes leave program results unchanged. So the metric captures detection of a specific kind of change, not oracle quality broadly. The authors acknowledge this, but readers should remember that OBCCG includes behavior-preserving changes that may never correspond to real faults. Also, 52.6% of detecting mutant–test pairs fail by crash rather than assertion failure; this inflates OBCC relative to what most developers would consider a test oracle. The authors report crash shares per project (24.8–76.0%) and verify that the RQ4 results hold regardless of crash inclusion, so this is handled reasonably. The 2% unresolved outcomes are a minor practical issue, not a conceptual flaw.
Who this is for: researchers and practitioners in software testing, especially those using Java and PIT. It gives an actionable way to separate coverage from detection, and the per-outcome records let developers inspect why a specific outcome is undetected.
I'd recommend accepting for peer review. The paper is methodologically rigorous, the central claim is supported by convergent evidence, and the limitations are honestly discussed. It advances the state of the art in test adequacy assessment beyond what slicing-based approaches can practically deliver.
The strongest objection
The paper defines, implements, and evaluates a new detection-based checked coverage metric (OBCC/OBCCG) with unusual thoroughness. The central claim—that Veritas identifies a meaningful gap between covered and detected behavior—rests on multiple lines of convergent evidence: (1) the qualitative sample confirming the gap's nature, (2) the asymmetric disagreement with slicing-based checked coverage, (3) the monotonic relationship between OBCCG and fault detection under assertion weakening (Kendall's τ = -1.00), and (4) the spatial association between gap-controlled code locations and surviving independent faults (12.4–54.7 percentage point difference). The authors explicitly acknowledge the key limitations: forced mutations are a proxy, OBCC measures detection not assertion quality, crashes count as detection, the association is not causation, and external validity is bounded by the 11 open-source Java projects. They validate the proxy with held-out faults using different mutation operators, avoiding circularity. The construct validity threat about crash-based detection inflating OBCC is addressed by reporting crash shares (52.6% overall, per-project ranges) and by both experiments in RQ4 showing that OBCCG tracks fault detection regardless of crash inclusion. The attribution mechanism (first differing trace entry) is sound for the single-decision-per-mutant design. No internally inconsistent assumption or unacknowledged flaw undermines the central claim; the paper's limitations are stated transparently and its evidence stack is robust.
The reader correctly identified that the main concern is the validity of forced outcome changes as a proxy for faults, noting that a third of activated-but-undetected outcomes leave results unchanged. This concern is acknowledged by the authors and addressed via RQ4 using held-out faults with different operators. The stress-test pass found no additional load-bearing concern that would shift the verdict. The paper's evidence is convergent, its limitations are transparent, and the internal logic is sound. The proposed concrete test further validates that the spatial association is not an artifact of crash-based detection, which would address a minor residual threat without altering the current 'ACCEPT' verdict.
The referee's report
The paper makes a strong empirical contribution by defining and validating a slicing-free, mutation-based metric for checked coverage at the branch-outcome level. The central claim—that structural coverage alone overstates test effectiveness—is supported by convergent evidence from multiple experiments, including a monotonic relationship with fault detection under assertion weakening (Kendall's τ = -1.00), asymmetric disagreement with slicing-based checked coverage, and a spatial association with surviving independent faults. The method is practically deployable (runs on all 11 projects, uses familiar mutation infrastructure) and the authors explicitly acknowledge key limitations (mutation proxy, crash-as-detection, unresolved outcomes, associational nature of RQ4). The thoroughness of the evaluation and the actionable nature of the gap classification (missing activation vs. missing detection) make this a significant advance.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that standard code coverage is misleading because it only reports which code runs, not whether the tests would notice a fault there. The authors introduce Veritas, a technique that replaces inferred dependence with direct experimental evidence: for each covered branch outcome, Veritas forces the opposite outcome, reruns the relevant tests, and counts the outcome as checked only if a previously passing test fails and the failure can be attributed to that outcome. Applying Veritas to 51,382 object-branch outcomes across 11 Java projects, they find that 18.2% of covered outcomes can be changed without any test failing—the behavior is executed but not detected. This gap is not a curiosity; it correlates with weaker fault detection: weakening assertions increases the gap and reduces held-out fault detection on every project, and fault sites near such gap outcomes are 12.4–54.7 percentage points more likely to have a surviving independent fault. The paper thus establishes that coverage alone hides important testing weaknesses and that a detection-based metric can expose them without expensive dynamic slicing.
What carries the argument. The central mechanism is a controlled mutation experiment on each covered branch outcome. For every binary conditional jump in Java bytecode (object-branch decision), Veritas constructs a mutant that forces the opposite outcome on every execution. It then reruns only the tests that reached that decision in the original program, compares clean and mutant traces, and attributes a check only when (1) the mutant changes the path at the decision (activation), (2) a previously passing test fails (detection), and (3) the first differing trace entry identifies the outcome (attribution). This replaces dynamic slicing with direct detection evidence. The paper instantiates this over object branch coverage (OBC), a compiled-code criterion comparable to source-level MC/DC. The resulting metrics are object branch coverage (OBC), object branch checked coverage (OBCC), and their gaps (OBCG, OBCCG), all expressed as shares of the total outcome set.
Core claim. Using Veritas on 51,382 object-branch outcomes from 11 open-source Java projects, the authors discover that 18.2% of covered outcomes are activated but undetected: forcing the opposite outcome changes execution but no test fails. This gap exists in every project and is concentrated in a small number of classes; even methods with full outcome coverage have at least one such outcome in 33.2% of cases. The paper further shows that this measured gap predicts weaker fault detection: a one-point increase in the object branch checked coverage gap (OBCCG) corresponds to a 0.62–1.57 percentage-point decrease in held-out fault detection, and fault sites near gap outcomes are 12.4–54.7 percentage points more likely to contain an undetected independent fault. Compared with slicing-based checked coverage, Veritas identifies 4,763 cases where tests detect a change that slicing does not credit, while 19% of outcomes slicing considers checked are not detected when changed. The authors conclude that directly testing outcomes provides actionable evidence of oracle effectiveness beyond structural coverage.
The premise it rests on. The paper assumes that forcing a branch outcome to its opposite via mutation is a valid proxy for a fault at that location, and that the resulting test failure evidence captures all relevant detection gaps; the authors validate this correlation with held-out mutants, but acknowledge that some realistic faults may not correspond to such a flip and that about a third of sampled gap outcomes leave program results unchanged, meaning the gap may overstate detection weaknesses in some cases.
If the paper is right (Pith's inferences, not the authors')
Testers can use Veritas to identify specific covered branch outcomes that lack detection evidence, moving from aggregate coverage scores to actionable inspection targets.
The approach refines mutation testing reports: surviving mutants are split into those never activated (needing different inputs) and those activated but undetected (needing stronger checks or different inputs), addressing different test weaknesses.
Automated test generation tools can target uncovered or activated-but-undetected outcomes with new inputs or stronger assertions, guided by Veritas's outcome-level evidence.
In high-assurance settings that require MC/DC (e.g., DO-178C Level A), Veritas offers a practical slicing-free way to measure whether covered outcomes are actually checked, potentially improving oracle quality assessment.
The concentrated gap (few classes hold most undetected outcomes) suggests that inspection effort can be prioritized effectively.
Veritas's reliance on forcing the opposite outcome may undercount detection for faults that do not correspond to a branch-flip, such as arithmetic errors that change a value without altering control flow; a more comprehensive approach might combine Veritas with other mutation operators.
The paper's validation uses mutants as proxies for faults; a direct study on real faults (e.g., from bug databases) could strengthen or qualify the claim that gap outcomes predict undetected real faults.
Veritas could be extended to other coverage criteria (e.g., source-level decisions, paths) by defining appropriate outcome-forcing mutations for those criteria, potentially generalizing the detection-based checking concept.
The fact that 52.6% of detecting pairs fail by crash rather than assertion suggests that assertion quality is one part of detection; Veritas measures detection broadly, so OBCC should be interpreted as a detection metric, not a pure oracle-quality metric.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Code coverage reports what tests execute, not whether tests detect faults. Checked coverage addresses this gap by tracing code that influences test oracles through dynamic slicing, but slicing is costly, and Java slicers struggle with modern language features and complex programs. We present Veritas, a slicing-free approach that directly tests whether covered behavior is checked. We target object branch coverage (OBC), a compiled-code structural criterion with fault-detection effectiveness comparable to source-level Modified Condition/Decision Coverage (MC/DC), which is required at the highest assurance level of DO-178C. For each covered object-branch outcome, Veritas forces the opposite outcome and credits checking only when the change is executed, a previously passing test fails, and the failure is attributable to that outcome. Thus, Veritas distinguishes execution from demonstrated fault detection without dynamic dependence analysis. We evaluated Veritas on eleven open-source Java projects containing 51,382 object-branch outcomes. We find that 18% of covered outcomes can be changed without failing any test, exposing weaknesses hidden by coverage. Among surviving mutants, about half are never activated by test inputs, while the rest are activated but undetected. Compared with slicing-based checked coverage, Veritas identifies 4,763 cases where tests detect a change that slicing does not credit, while 19% of cases credited by slicing produce no detected failure when changed. Weakening assertions increases this gap and reduces fault detection in every project. Moreover, code locations controlled by weakly checked outcomes are 12.4–54.7 percentage points more likely to have an undetected independent fault than locations controlled by checked outcomes. These findings show that directly testing outcomes provides actionable evidence of oracle effectiveness beyond structural coverage.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-27
The authors' main claim, as the classifier picked it out 1.00
“These findings show that directly testing outcomes provides actionable evidence of oracle effectiveness beyond structural coverage.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.27 · human participants 0.02 · controlled comparison 0.74 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.05
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Code coverage reports what tests execute, not whether tests detect faults. | background 0.93 | 0.05 | 0.05 | asserted 1.00 |
| Checked coverage addresses this gap by tracing code that influences test oracles through dynamic slicing, but slicing is costly, and Java slicers struggle with modern language features and complex programs. | background 0.89 | 0.05 | 0.03 | asserted 1.00 |
| We present Veritas, a slicing-free approach that directly tests whether covered behavior is checked. | contribution 1.00 | 0.86 | 0.12 | asserted 0.99 |
| We target object branch coverage (OBC), a compiled-code structural criterion with fault-detection effectiveness comparable to source-level Modified Condition/Decision Coverage (MC/DC), which is required at the highest assurance level of DO-178C. | contribution 0.59 | 0.25 | 0.06 | asserted 1.00 |
| For each covered object-branch outcome, Veritas forces the opposite outcome and credits checking only when the change is executed, a previously passing test fails, and the failure is attributable to that outcome. | method 0.99 | 0.86 | 0.09 | asserted 0.98 |
| Thus, Veritas distinguishes execution from demonstrated fault detection without dynamic dependence analysis. | method 0.52 | 0.59 | 0.18 | asserted 1.00 |
| We evaluated Veritas on eleven open-source Java projects containing 51,382 object-branch outcomes. | method 0.76 | 0.97 | 0.03 | asserted 0.98 |
| We find that 18% of covered outcomes can be changed without failing any test, exposing weaknesses hidden by coverage. | result 1.00 | 0.97 | 0.16 | asserted 1.00 |
| Among surviving mutants, about half are never activated by test inputs, while the rest are activated but undetected. | result 1.00 | 0.93 | 0.05 | asserted 0.99 |
| Compared with slicing-based checked coverage, Veritas identifies 4,763 cases where tests detect a change that slicing does not credit, while 19% of cases credited by slicing produce no detected failure when changed. | result 1.00 | 0.95 | 0.15 | asserted 1.00 |
| Weakening assertions increases this gap and reduces fault detection in every project. | result 0.99 | 0.90 | 0.07 | asserted 1.00 |
| Moreover, code locations controlled by weakly checked outcomes are 12.4--54.7 percentage points more likely to have an undetected independent fault than locations controlled by checked outcomes. | result 0.97 | 0.92 | 0.13 | asserted 0.93 |
| These findings show that directly testing outcomes provides actionable evidence of oracle effectiveness beyond structural coverage. | interpretation 1.00 | 0.67 | 0.87 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 1.00
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-27: 1 request, 9,505 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
References (42)
- [1] Hiralal Agrawal and Joseph R. Horgan. 1990. Dynamic program slicing.SIGPLAN Not.25, 6 (June 1990), 246–256. doi:10.1145/93548.93576 on Pithdoi:10.1145/93548.93576
- [2] Khaled Ahmed, Mieszko Lis, and Julia Rubin. 2021. Slicer4J: a dynamic slicer for Java(ESEC/FSE 2021). Association for Computing Machinery, New York, NY, USA, 1570–1574. doi:10.1145/3468264.3473123 on Pithdoi:10.1145/3468264.3473123
- [3] Toufique Ahmed, Premkumar Devanbu, Christoph Treude, and Michael Pradel. 2025. Can LLMs Replace Manual Annotation of Software Engineering Artifacts?. In2025 IEEE/ACM 22nd International Conference on Mining Software Repositories (MSR)(Ottawa, ON, Canada). IEEE Press, 526–538. d... on Pithdoi:10.1109/msr66628.2025.00086
- [4] Paul Ammann and Jeff Offutt. 2016.Introduction to Software Testing(2nd ed.). Cambridge University Press on Pith
- [5] Eric Bruneton, Romain Lenglet, and Thierry Coupaye. 2002. ASM: A Code Manipulation Tool to Implement Adaptable Systems. InAdaptable and Extensible Component Systems. Grenoble, France on Pith
- [6] Tae Joon Byun, Vaibhav B. Sharma, Sanjai Rayadurgam, Stephen A. McCamant, and Mats P. E. Heimdahl. 2017. Toward Rigorous Object-Code Coverage Criteria. In2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 328–338. doi:10.1109/ISSRE.2017.33 on Pithdoi:10.1109/issre.2017.33
- [7] John Joseph Chilenski and Steven P. Miller. 1994. Applicability of Modified Condition/Decision Coverage to Software Testing.Software Engineering Journal9, 5 (1994), 193–200. doi:10.1049/sej.1994.0025 on Pithdoi:10.1049/sej.1994.0025
- [8] Henry Coles, Thomas Laurent, Christopher Henard, Mike Papadakis, and Anthony Ventresque. 2016. PIT: a practical mutation testing tool for Java (demo). InProceedings of the 25th International Symposium on Software Testing and Analysis(Saarbrücken, Germany)(ISSTA 2016). Associat... on Pithdoi:10.1145/2931037.2948707
- [9] R.A. DeMillo, R.J. Lipton, and F.G. Sayward. 1978. Hints on Test Data Selection: Help for the Practicing Programmer. Computer11, 4 (1978), 34–41. doi:10.1109/C-M.1978.218136 on Pithdoi:10.1109/c-m.1978.218136
- [10] EclEmma. 2024. JaCoCo Java Code Coverage Library. https://www.jacoco.org/jacoco/. Accessed: 2026-07-31 on Pith
- [11] FasterXML. 2024. Jackson Core. https://github.com/FasterXML/jackson-core. Version 2.17.2. Accessed: 2026-09-30 on Pith
- [12] Clemens Hammacher. 2008. Design and Implementation of an Efficient Dynamic Slicer for Java. Bachelor’s Thesis on Pith
- [13] Clemens Hammacher. 2022.JavaSlicer. https://github.com/hammacher/javaslicer Open-source dynamic slicing tool for Java on Pith
- [14] Jonathan Hedley. 2024. jsoup: Java HTML Parser. https://github.com/jhy/jsoup. Version 1.18.1. Accessed: 2026-09-30 on Pith
- [15] Soneya Binta Hossain, Matthew B. Dwyer, Sebastian Elbaum, and Anh Nguyen-Tuong. 2023. Measuring and Mitigating Gaps in Structural Testing. InProceedings of the 45th International Conference on Software Engineering(Melbourne, Victoria, Australia)(ICSE ’23). IEEE Press, 1712–172... on Pithdoi:10.1109/icse48619.2023.00147
- [16] W. E. Howden. 1982. Weak Mutation Testing and Completeness of Test Sets.IEEE Trans. Softw. Eng.8, 4 (July 1982), 371–379. doi:10.1109/TSE.1982.235571 on Pithdoi:10.1109/tse.1982.235571
- [17] Laura Inozemtseva and Reid Holmes. 2014. Coverage is not strongly correlated with test suite effectiveness. In Proceedings of the 36th International Conference on Software Engineering(Hyderabad, India)(ICSE 2014). Association for Computing Machinery, New York, NY, USA, 435–445... on Pithdoi:10.1145/2568225.2568271
- [18] Kush Jain, Goutamkumar Tulajappa Kalburgi, Claire Le Goues, and Alex Groce. 2023. Mind the Gap: The Difference Between Coverage and Mutation Score Can Guide Testing Efforts. In2023 IEEE 34th International Symposium on Software Reliability Engineering (ISSRE). 102–113. doi:10.1... on Pithdoi:10.1109/issre59848.2023.00036
- [19] JFree. 2025. JFreeChart. https://github.com/jfree/jfreechart. Version 1.5.6. Accessed: 2026-09-30 on Pith
- [20] JFree. 2025. JFreeSVG. https://github.com/jfree/jfreesvg. Version 5.0.7. Accessed: 2026-09-30 on Pith
show 22 more
- [21] Joda.org. 2024. Joda-Convert. https://github.com/JodaOrg/joda-convert. Version 3.0.1. Accessed: 2026-09-30 on Pith
- [22] René Just, Darioush Jalali, Laura Inozemtseva, Michael D. Ernst, Reid Holmes, and Gordon Fraser. 2014. Are mutants a valid substitute for real faults in software testing?. InProceedings of the 22nd ACM SIGSOFT International Symposium on Foundations of Software Engineering(Hong... on Pithdoi:10.1145/2635868.2635929
- [23] Megan Maton, Gregory M. Kapfhammer, and Phil McMinn. 2025. Where Tests Fall Short: Empirically Analyzing Oracle Gaps in Covered Code. InProceedings of the ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE, 1–11. doi:10.1109/ESEM641... on Pithdoi:10.1109/esem64174.2025.00063
- [24] Rainer Niedermayr, Elmar Juergens, and Stefan Wagner. 2016. Will my tests tell me if I break this code?. InProceedings of the International Workshop on Continuous Software Evolution and Delivery(Austin, Texas)(CSED ’16). Association for Computing Machinery, New York, NY, USA, ... on Pithdoi:10.1145/2896941.2896944
- [25] A. J. Offutt and S. D. Lee. 1994. An Empirical Evaluation of Weak Mutation.IEEE Trans. Softw. Eng.20, 5 (May 1994), 337–344. doi:10.1109/32.286422 on Pithdoi:10.1109/32.286422
- [26] Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. 2019. Chapter Six - Mutation Testing Advances: An Analysis and Survey. Advances in Computers, Vol. 112. Elsevier, 275–378. doi:10.1016/bs.adcom. 2018.03.015 , Vol. 1, No. 1, Article . Publicati... on Pithdoi:10.1016/bs.adcom
- [27] RTCA. 2011.DO-178C: Software Considerations in Airborne Systems and Equipment Certification. Technical Report. RTCA, Inc., Washington, DC, USA on Pith
- [28] David Schuler and Andreas Zeller. 2011. Assessing Oracle Quality with Checked Coverage(ICST ’11). IEEE Computer Society, USA, 90–99. doi:10.1109/ICST.2011.32 on Pithdoi:10.1109/icst.2011.32
- [29] David Schuler and Andreas Zeller. 2013. Checked coverage: an indicator for oracle quality.Software Testing, Verification and Reliability23, 7 (2013), 531–551. doi:10.1002/stvr.1497 on Pithdoi:10.1002/stvr.1497
- [30] Amanda Schwartz, Daniel Puckett, Ying Meng, and Gregory Gay. 2018. Investigating faults missed by test suites achieving high code coverage.J. Syst. Softw.144, C (Oct. 2018), 106–120. doi:10.1016/j.jss.2018.06.024 on Pithdoi:10.1016/j.jss.2018.06.024
- [31] August Shi, Jonathan Bell, and Darko Marinov. 2019. Mitigating the effects of flaky tests on mutation testing. In Proceedings of the 28th ACM SIGSOFT International Symposium on Software Testing and Analysis(Beijing, China)(ISSTA 2019). Association for Computing Machinery, New ... on Pithdoi:10.1145/3293882.3330568
- [32] The Apache Software Foundation. 2024. Apache Commons CLI. https://github.com/apache/commons-cli. Version 1.9.0. Accessed: 2026-09-30 on Pith
- [33] The Apache Software Foundation. 2024. Apache Commons Codec. https://github.com/apache/commons-codec. Version 1.17.1. Accessed: 2026-09-30 on Pith
- [34] The Apache Software Foundation. 2024. Apache Commons CSV. https://github.com/apache/commons-csv. Version 1.11.0. Accessed: 2026-09-30 on Pith
- [35] The Apache Software Foundation. 2025. Apache Commons JXPath. https://github.com/apache/commons-jxpath. Version 1.4.0. Accessed: 2026-09-30 on Pith
- [36] The Apache Software Foundation. 2026. Apache Commons Validator. https://github.com/apache/commons-validator. Version 1.11.0. Accessed: 2026-09-30 on Pith
- [37] The JUnit Team. 2024. JUnit 5. https://junit.org/junit5/. Accessed: 2026-09-24 on Pith
- [38] ThreeTen. 2026. ThreeTen-Extra. https://github.com/ThreeTen/threeten-extra. Version 1.10.0. Accessed: 2026-09-30 on Pith
- [39] Oscar Luis Vera-Pérez, Martin Monperrus, and Benoit Baudry. 2018. Descartes: a PITest engine to detect pseudo-tested methods: tool demonstration. InProceedings of the 33rd ACM/IEEE International Conference on Automated Software Engineering(Montpellier, France)(ASE ’18). Associ... on Pithdoi:10.1145/3238147.3240474
- [40] Mark Weiser. 1981. Program slicing. InProceedings of the 5th International Conference on Software Engineering(San Diego, California, USA)(ICSE ’81). IEEE Press, 439–449 on Pith
- [41] Edwin B. Wilson. 1927. Probable Inference, the Law of Succession, and Statistical Inference.J. Amer. Statist. Assoc.22, 158 (1927), 209–212. doi:10.1080/01621459.1927.10502953 on Pithdoi:10.1080/01621459.1927.10502953
- [42] Yucheng Zhang and Ali Mesbah. 2015. Assertions are strongly correlated with test suite effectiveness. InProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering(Bergamo, Italy)(ESEC/FSE 2015). Association for Computing Machinery, New York, NY, USA, 214... on Pithdoi:10.1145/2786805.2786858
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment