REVIEW 4 major objections 4 minor 14 references
Testability First!
T0 review · 4 major / 4 minor · reviewed 2026-08-14 · deepseek-v4-flash
Pith's one-line read This paper hypothesizes that poor software test quality is rooted in low testability, which must be planned for early in development and supported by lifecycle-wide testing policies.
desk verdict An honest, hypothesis-generating industrial case study that usefully points at testability as an overlooked variable in TDD/TLD comparisons, but its quantitative evidence for the causal claim is weak and should not be oversold. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is testability, defined as 'ease of testing': the effort needed to write tests that are effective at revealing defects and manageable in the face of change. That definition carries the argument because the observed behavior—developers abandon tests when effort is high—links test quality back to design characteristics such as single responsibility, number of dependencies, and the existence of exact behavioral specifications. The hypothesized mechanism is the causal chain from early attention to testing, to improved testability, to lower testing effort, to higher-quality tests, and finally to fewer bugs.
What would settle it
On the paper's own data, re-analyze the 244 buggy components with a regression that controls for lines of code and dependency count; if tested components no longer show fewer bugs, the claimed advantage is not testability but component size. More generally, run a matched experiment in several projects where half the components receive explicit design-time testability reviews before implementation; if that half does not show higher mutation scores and lower post-release bug rates, the testability-first hypothesis is unsupported.
Extended reading notes
Core claim
The central claim is the testability-first hypothesis: regardless of whether tests are written before or after production code, a component will only receive high-quality tests if it is designed for testing, and that design must come from early attention to testing and from policies that apply across the lifecycle. The supporting evidence comes from one industrial case: of 244 components affected by the 200 analyzed bugs, only eight were covered by unit tests; tested components had fewer bugs than untested ones; developers wrote tests in only 2% of the 140 cases classified as hard to test; and tests added after the coverage policy was introduced achieved significantly higher mutation scores than older tests. The paper also claims that conflicting outcomes in the test-driven-development literature look less contradictory once testability differences between projects are recognized as a missing variable.
Load-bearing premise
The load-bearing premise is that early attention to testing improves test quality through testability itself; the paper's own data show the open-bug count staying roughly flat after the testing policy began, and its comparisons do not control for component size or complexity.
Editorial extensions
If this is right
- Comparisons of test-first and test-later development should measure or match testability, or their conclusions about which practice is better will be confounded.
- Teams that want better tests should schedule testability reviews and testing policies at the start of a component's life, not after the code is written.
- Components with many dependencies, unclear behavior, or multiple responsibilities will tend to be untested, so improving testability is a prerequisite to raising coverage on those components.
- A management-imposed coverage goal can raise the quality of newly written tests even if the total number of open bugs stays roughly constant.
Reading between the lines
- If testability is the true driver, then simple structural metrics like dependency count and method size should predict which components have weak test suites before bugs are even counted; this could be checked directly on the paper's dataset.
- The paper's results do not separate the coverage policy from other simultaneous changes such as improved Scrum discipline and customer communication, so the improved mutation scores could partly reflect clearer requirements rather than testability itself.
- Adopting the testability-first view would suggest adding a design-time gate: no component is accepted for implementation until testers confirm it can be exercised through a public interface without special scaffolding.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports an industrial case study of a large Swiss logistics platform (XYZ), analyzing bug data, test coverage, and developer interviews over a five-year period. The authors observe low test adoption and low test effectiveness, and identify testability (the ease of writing effective tests) as a neglected factor. On the basis of the observations and interviews, they propose a 'testability first' hypothesis: establishing clear testing policies throughout the development lifecycle and giving early attention to testing each component are crucial for achieving high testability of the code base and thus high-quality tests. The paper also reports a quantitative comparison showing that the eight tested components had fewer bugs than the 236 untested components, and mutation analysis indicating that newer tests (post-2017) are more effective.
Significance. If the testability-first hypothesis is correct, it would provide a valuable explanation for contradictory findings in test-driven development versus test-later studies by identifying testability as a key confounder. The paper's strengths include access to a real industrial system with five years of history, a mixed-methods approach combining quantitative bug/test analysis with interviews, and an explicit attempt to use mutation testing to assess test quality. The authors also clearly acknowledge the tentative nature of their central claim by framing it as a hypothesis. However, the empirical evidence for the hypothesis is weak: the quantitative comparison is confounded, the 'hard to test' classification is subjective, and the temporal evidence does not show a reduction in open bugs after the policy introduction. The hypothesis is plausible but not established by this study. The paper is best viewed as an exploratory case study that generates a testable hypothesis for future multi-project research.
major comments (4)
- [III-A] The central quantitative support for the claim that tested components are less bug-prone is a comparison between 8 tested and 236 untested components. This comparison is confounded: the two groups differ systematically in size, complexity, age, and centrality, all of which are likely related to both testability and bug-proneness. The paper reports no metrics on these characteristics, so the observed association is equally consistent with 'easy-to-test components are more likely to be tested' as with 'testing reduces bugs'. This is a load-bearing issue because the paper explicitly states 'Tested components had fewer bugs than other components' (Section III-A). To support the claim, the authors should at minimum report descriptive statistics for the two groups (e.g., lines of code, number of methods, change churn) and, ideally, perform a regression or matched comparison controlling for these confounders, or temper the causal language to a clearly labeled exploratory observation.
- [II-B] The 'hard to test' classification is a key explanatory construct: it is used to argue that untested components were not tested because they were hard to test. However, the classification was made by the authors together with one senior developer after inspecting the buggy code and, apparently, with knowledge of whether tests existed. This risks an ex post rationalization: a component that is untested may be labeled 'hard to test' simply because the absence of tests needs explanation. The paper should describe the classification procedure in more detail, ideally with an independent second rater and an inter-rater reliability measure, or at least with predefined, code-level criteria for 'hard to test' and a validation step that does not depend on the outcome being explained.
- [III-A and Figure 1] The paper's own Figure 1 shows no decrease in the number of open bugs after the introduction of the testing policy in 2017; indeed, the average number of open bugs slightly increased. The authors explain this by citing the team manager's 'cleanup' explanation, which is taken at face value and is not independently verified. This matters because the paper later claims that 'a deliberate management policy to improve test coverage led to an increase in tests and in the quality of the code' (Section VI). The only quantitative evidence for improved quality after the policy is a higher mutation score for newer tests, but this is not connected to the policy in a controlled way (e.g., before/after defect density per component). The manuscript should either provide a more rigorous analysis of the temporal relationship (e.g., comparing defect rates per unit of code or per component before and after, accounting for the cleanup explanation) or explicitly limit the claim to the observed increase in test quality without asserting an impact on the bug count.
- [IV] The central claim in the Discussion is framed as a hypothesis, which is appropriately cautious, but some sentences in the paper (e.g., 'we found that the root cause of this phenomenon is that the testability of the project is low' in Section IV, and 'Tested components appear to be less prone to bugs than untested ones' in Section III-A) use language that goes beyond what the evidence supports. Given that the causal chain from early attention to testing → high testability → high-quality tests → fewer bugs is not established, the paper should consistently separate observed facts, developer-reported explanations, and the authors' hypothesis, and avoid implying causation where only association or self-report is available.
minor comments (4)
- [II-B] The sentence 'We then inspected whether the code part that caused the bug was covered by a unit test' could be clearer: it should specify whether the coverage was measured at the time of the bug report or at the time of analysis, since tests may have been added between the bug report and the inspection.
- [III-A] Figure 2 would benefit from labeled axes in the text (the figure is clear in the image but the caption could mention the distribution explicitly). Additionally, the statement that 'about a quarter of the untested components suffered from more than five bugs' should be accompanied by the actual count or proportion, as it is not directly visible from the figure.
- [IV] The phrase 'we see a clear need to encourage developers to start testing early' is a reasonable recommendation, but it is a logical leap from the case study data. The paper could clarify that this is a recommendation for practice based on the hypothesis, not a proven result.
- [References] Reference [8] has typographical issues: 'Matja Panur' and 'Ciglari' are missing diacritics (should be Matjaž Pančur and Mojca Ciglarič). Also, the reference list should be checked for consistency in author name formatting.
Circularity Check
No significant circularity: the paper's central claim is an explicitly stated hypothesis, and its self-citations are incidental methodology references.
full rationale
The paper does not derive any quantity from a fitted model or from a self-citation chain. Its central claim, that early attention to testing and clear testing policies ensure high testability and thus high-quality tests, is explicitly framed as a hypothesis ('we hypothesize that ...') in Sections IV and VI, not as a forced consequence of the data. The quantitative observations (tested components had fewer bugs; hard-to-test components were rarely tested; mutation scores improved after 2017) are empirical associations and descriptive statistics, not predictions generated from the same inputs. No parameter is fitted and then renamed as a prediction; no equation equates an output to an input by construction. The paper's self-citations, [6] and [14], are used only to support the mechanical notion of focal methods and goal-oriented mutation testing in the analysis methodology; they do not carry the central testability-first argument, and the claim would stand or fall independently of them. The main weaknesses of the paper are validity concerns (confounded comparison of eight tested versus 236 untested components, subjective hard-to-test classification, reliance on the manager's 'cleanup' explanation), but these are threats to causal inference, not circularity. Accordingly, no circular step can be exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Representativeness of the 200-bug sample
- domain assumption Subjective classification of 'hard to test'
- domain assumption Interview responses reflect actual behavior
- domain assumption Testability defined as ease of testing
Cite this review
Pith. "Pith review of Testability First!." pith.science (2026). https://pith.science/paper/QRTSNCE7
@misc{pith2026190801476,
author = {Pith},
title = {Pith review of: Testability First!},
year = {2026},
howpublished = {\url{https://pith.science/paper/QRTSNCE7}},
note = {Machine review of arXiv:1908.01476}
}
read the original abstract
The pivotal role of testing in high-quality software production has driven a significant effort in evaluating and assessing testing practices. We explore the state of testing in a large industrial project over an extended period. We study the interplay between bugs in the project and its test cases, and interview developers and stakeholders to uncover reasons underpinning our observations. We realized that testing is not well adopted, and that testability (ie, ease of testing) is low. We found that developers tended to abandon writing tests when they assessed the effort to be high. Frequent changes in requirements and pressure to add new features also hindered developers from writing tests. Regardless of the debates on test first or later, we hypothesize that the underlying reasons for poor test quality are rooted in a lack of attention to testing early in the development of a software component, leading to poor testability of the component. However, testability is usually overlooked in research that studies the impact of testing practices, and should be explicitly taken into account.
Figures
Reference graph
Works this paper leans on
-
[1]
Developer testing in the IDE: Patterns, beliefs, and behavior
Moritz Beller, Georgios Gousios, Annibale Panichella, Sebastian Proksch, Sven Amann, and Andy Zaidman. Developer testing in the IDE: Patterns, beliefs, and behavior. IEEE Transactions on Software Engineering, 45(3):261–284, March 2019
work page 2019
-
[2]
Wilson Bissi, Adolfo Gustavo Serra Seca Neto, and Maria Claudia Figueiredo Pereira Emer. The effects of test driven development on internal quality, external quality and productivity: A systematic review. Information and Software Technology , 74:45 – 54, 2016
work page 2016
-
[3]
Borle, Meysam Feghhi, Eleni Stroulia, Russell Greiner, and Abram Hindle
Neil C. Borle, Meysam Feghhi, Eleni Stroulia, Russell Greiner, and Abram Hindle. Analyzing the effects of test driven development in GitHub. Empirical Software Engineering, 23(4):1931–1958, Aug 2018
work page 1931
-
[4]
Davide Fucci, Hakan Erdogmus, Burak Turhan, Markku Oivo, and Natalia Juristo. A dissection of the test-driven development process: Does it really matter to test-first or to test-last? IEEE Transactions on Software Engineering, 43(7):597–614, July 2017
work page 2017
-
[5]
A survey on software testability
Vahid Garousi, Michael Felderer, and Feyza Nur Klc ¸aslan. A survey on software testability. Information and Software Technology, 108:35 – 64, 2019
work page 2019
-
[6]
Automat- ically identifying focal methods under test in unit test cases
Mohammad Ghafari, Carlo Ghezzi, and Konstantino Rubinov. Automat- ically identifying focal methods under test in unit test cases. In 2015 IEEE 15th International Working Conference on Source Code Analysis and Manipulation (SCAM) , pages 61–70, Sep. 2015
work page 2015
-
[7]
On the relation between unit testing and code quality
Lucas Gren and Vard Antinyan. On the relation between unit testing and code quality. In 2017 43rd Euromicro Conference on Software Engineering and Advanced Applications (SEAA) , pages 52–56, Aug 2017
work page 2017
-
[8]
Impact of test-driven development on productivity, code and tests: A controlled experiment
Matja Panur and Mojca Ciglari. Impact of test-driven development on productivity, code and tests: A controlled experiment. Information and Software Technology, 53(6):557 – 573, 2011. Special Section: Best papers from the APSEC
work page 2011
Show all 14 references
-
[9]
Chapter six — Mutation testing advances: An analysis and survey
Mike Papadakis, Marinos Kintis, Jie Zhang, Yue Jia, Yves Le Traon, and Mark Harman. Chapter six — Mutation testing advances: An analysis and survey. volume 112 of Advances in Computers , pages 275 – 378. Elsevier, 2019
2019
-
[10]
Practitioners’ views on good software testing practices
Kochhar Pavneet Singh, Xin Xia, and David Lo. Practitioners’ views on good software testing practices. In Proceedings of the 41st International Conference on Software Engineering , ICSE ’19, 2019
2019
-
[11]
How effectively is defective code actually tested?: An analysis of JUnit tests in seven open source systems
Jean Petri ´c, Tracy Hall, and David Bowes. How effectively is defective code actually tested?: An analysis of JUnit tests in seven open source systems. In Proceedings of the 14th International Conference on Predic- tive Models and Data Analytics in Software Engineering , PROM...
2018
-
[12]
Investigating faults missed by test suites achieving high code coverage
Amanda Schwartz, Daniel Puckett, Ying Meng, and Gregory Gay. Investigating faults missed by test suites achieving high code coverage. Journal of Systems and Software , 144:106 – 120, 2018
2018
-
[13]
Predicting different levels of the unit testing effort of classes using source code metrics: a multiple case study on open-source software
Fadel Toure, Mourad Badri, and Luc Lamontagne. Predicting different levels of the unit testing effort of classes using source code metrics: a multiple case study on open-source software. Innovations in Systems and Software Engineering , 14(1):15–46, Mar 2018
2018
-
[14]
Goal-oriented mutation testing with focal methods
Sten Vercammen, Mohammad Ghafari, Serge Demeyer, and Markus Borg. Goal-oriented mutation testing with focal methods. In Proceedings of the 9th ACM SIGSOFT International Workshop on Automating TEST Case Design, Selection, and Evaluation , A-TEST 2018, pages 23–30, New York, NY ...
2018
Reviewed August 14, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.