REVIEW 4 major objections 5 minor 4 references
Evolution analysis of software quality metrics in an open-source java project: A case study on TestNG
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read Over five releases, TestNG's cyclomatic complexity and lack of cohesion declined while its size stayed flat, and the paper reads this as a maturing, more maintainable framework.
desk verdict Replication data and a clean setup can't save a case study whose own statistical table contradicts its central maintainability claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is a longitudinal metric matrix: eleven Understand-extracted object-oriented metrics, one row per class, observed at five points in TestNG's history. The two indicators doing the most argumentative work are MaxCyclomatic (branching complexity per method) and PercentLackOfCohesion (LCOM, a 0-100 measure of whether a class's methods share related state), because those are the metrics that decline while the codebase stays roughly constant in size. The statistical engine is the Wilcoxon signed-rank test, a non-parametric paired test comparing matched classes between two versions; its W statistic is the smaller of the summed ranks of positive and negative differences. A NaN result is treated as no evidence of change, either because the metric was constant across classes or because data rows were removed during cleaning.
What would settle it
Re-run the extraction on the same five source snapshots and repeat the Wilcoxon tests on a fully documented matching table that includes added, removed, and renamed classes. If mean and median cyclomatic complexity rise rather than fall when the full class population is used, or if the matched set is a shrinking minority of all classes, the stabilization conclusion is not supported.
Extended reading notes
Core claim
On the paper's own terms, its contribution is a metric-level profile of TestNG's evolution: over the sampled versions the framework grew in test code volume only mildly, while mean cyclomatic complexity fell from 2.12 to 1.58, LCOM fell from 16.09 to 9.68, and declared and instance method counts dropped after 6.13.1. Pairwise Wilcoxon tests found statistically significant differences in LOC between the older versions but no detectable difference between the newest release and its predecessors; the author interprets this plateau as stabilization. In the same statistical pass, coupling, inheritance depth, and related class-level metrics returned NaN in most comparisons, and the paper treats that as no significant change due to constant values or data gaps. The conclusion is that TestNG has evolved into a more stable and maintainable framework through sustained development, consistent refactoring, and thoughtful architectural improvements.
Load-bearing premise
The whole analysis rests on the assumption that classes can be matched meaningfully across versions and that dropping rows with null or all-zero metric values leaves a representative sample; if classes were renamed, added, or removed often, the surviving matched classes may not represent the codebase's real evolution.
Editorial extensions
If this is right
- Maintainers can use version-paired Wilcoxon tests on metric snapshots to tell whether a release actually changed a quality dimension, rather than relying on raw averages.
- The stable LOC with falling complexity implies new functionality was absorbed through simplification, not just added code, so the project can continue growing without obvious bloat.
- The lack of significant differences for version 7.11 versus earlier versions means the latest release window was structurally quiet; changes in that period should be lower risk to downstream users.
- Unchanged coupling and inheritance metrics suggest TestNG's public class architecture remained stable even as internal methods were refactored, a favorable property for a widely used library.
- The protocol generalizes beyond TestNG: periodic metric extraction plus a paired non-parametric test can be embedded in the quality assurance of any evolving object-oriented codebase.
Reading between the lines
- Beyond the paper: if the down-then-flat metric pattern is a general maturity signature, then a version whose LCOM and cyclomatic values stop falling may mark the transition from active refactoring to maintenance mode, and metric monitoring should treat that plateau as reassurance rather than alarm.
- Beyond the paper: the same protocol could be turned into a cheap regression gate, comparing touched classes' LCOM and cyclomatic values to a previous release and flagging increases before merge.
- Beyond the paper: applying the same measurement recipe to comparable open-source testing libraries would show whether TestNG's trajectory is a general pattern of mature software or a project-specific story.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents a longitudinal case study of the TestNG open-source Java testing framework, analyzing five versions (5.13, 6.0.1, 6.13.1, 7.5, 7.11.0) with the Understand static analysis tool. Eleven object-oriented metrics are extracted, visualized, and tested for change across versions using Wilcoxon signed-rank tests. The paper's central claim is that TestNG has evolved into a more stable and maintainable framework, as evidenced by reduced cyclomatic complexity, improved cohesion, and structural streamlining. The authors also offer research-question answers and recommendations for maintainers of similar projects.
Significance. If the analysis were sound, this would be a modest but useful case study on metric evolution in a widely used open-source testing framework. The paper has some strengths: the dataset and analysis scripts are promised to be publicly available, the non-parametric Wilcoxon test is an appropriate choice for paired, non-normal metric data, and the research questions are clearly stated. However, the central conclusion is directly contradicted by the paper's own statistical results, and key data-curation decisions are unreported. As presented, the manuscript does not provide reliable evidence for the claimed improvement in maintainability, so the significance of the contribution is not realized.
major comments (4)
- [§4.1, §4.4, Table 4, Conclusion] The central claim of improved maintainability is undermined by an internal contradiction. Section 4.1 states under 'Complexity Reduction' that Cyclomatic Complexity showed a downward trend, and the Conclusion repeats that reductions in cyclomatic complexity support the maintainability claim. Yet Section 4.4 and Table 4 report that the Wilcoxon signed-rank test for MaxCyclomaticComplexity comparing versions 5.13 and 7.11 is significant with p < 0.00001, with the interpretation 'Indicates increased complexity over time.' These two statements cannot both describe the same metric. Either the descriptive means in Table 2 are not comparable to the matched subset used for the Wilcoxon test, or the inferential result is misreported. Because the paper's main conclusion is inferred from this trend, the central claim is not supported by the evidence as presented.
- [§4.4, Table 3] The LOC results for version 7.11 against every other version report T-value and P-value as NaN and are interpreted as 'No significant difference detected' or 'No significant difference'. NaN is not a valid outcome of the Wilcoxon signed-rank test; it indicates that the test was not computed, typically because all differences were zero or data were missing. Consequently, the rows in Table 3 provide no statistical evidence of stabilization, and the conclusion's statement that later versions 'exhibited greater consistency and fewer structural changes' is unsupported by this table.
- [§3.4, §4.4] The paper never reports how many classes existed in each version, how many classes were added, removed, or renamed across versions, or how many matched pairs remained after removing null and all-zero rows. Section 3.4 says only that data were filtered using Excel PivotTables 'to remove null values and ensure consistency in class-level comparisons,' and Section 4.4 repeats that rows with all-zero or null values were excluded. If the matched subset is biased toward classes that survive across versions, then the significant increase in MaxCyclomaticComplexity and the frequent NaN results could be artifacts of data curation rather than evidence about the codebase as a whole. This is load-bearing because the Wilcoxon tests are the only statistical validation of the descriptive trends.
- [§4.4, Table 4, §4.1, Table 2] The treatment of PercentLackOfCohesion is inconsistent. Section 4.1 reports 'Improved Cohesion' based on a steady decrease in LCOM, and Table 2 shows LCOM falling from 16.09 in version 5.13 to 9.68 in version 7.11.0. However, Table 4 reports the Wilcoxon result for PercentLackOfCohesion as NaN with the interpretation 'Metric unchanged or data missing.' A NaN result cannot confirm either 'unchanged' or 'improved'; the paper must explain why the paired test failed and whether the descriptive decrease remains after removing null and constant rows.
minor comments (5)
- [Table 2] The row order lists testng-7.11.0 immediately before testng-7.5, which breaks the stated chronological ordering and the narrative that treats 7.5 as earlier than 7.11.0; reorder the rows so testng-7.5 appears before testng-7.11.0.
- [Conclusion] The conclusion refers to 'cyclical complexity' instead of 'cyclomatic complexity'; fix this typo.
- [§3.5] Section 3.5 contains an incomplete sentence: 'All extracted metrics for different versions of Dataset 4 scripts and notebooks used for data processing and statistical analysis are available in the GitHub repository 5.' Rewrite this to identify the dataset and repository clearly and to list the provided artifacts.
- [Table 1] The metric labels in Table 1 mix Understand column names with classic object-oriented metric definitions; in particular, RFC is described as 'Total number of methods per class' and WMC as 'Weighted methods per class' while the Understand labels are CountDeclMethodAll and CountDeclInstanceMethod. Please clarify the exact definitions used so readers can interpret the values in Table 2.
- [Title and Abstract] The title has formatting artifacts such as 'OPEN -SOURCE' and 'C ASE STUDY'; these should be cleaned up before publication.
Circularity Check
No derivation-level circularity; the study is an empirical measurement with an internal inconsistency, not an input-output circularity.
full rationale
This paper is a measurement and statistical-analysis case study rather than a derivation. It extracts eleven object-oriented metrics from five TestNG versions using the Understand tool, applies the Wilcoxon signed-rank test, and interprets the observed trends as evidence of improved maintainability. There is no fitted parameter subsequently renamed as a prediction, no equation whose output is identical to its input, no self-citation chain, and no uniqueness theorem imported from prior work. The conclusion that TestNG 'has evolved into a more stable and maintainable framework' is an interpretive judgment placed on the metric values, but the values themselves come from an external static-analysis tool and are not constructed to entail that conclusion. The most notable weakness is internal: Section 4.1 reports a downward trend in cyclomatic complexity, while Section 4.4 and Table 4 report a statistically significant increase in MaxCyclomaticComplexity between 5.13 and 7.11. This is a contradiction in the presentation and interpretation of results, and it undermines the support for the maintainability claim, but it is not a circular reduction of the claim to its own inputs. Similarly, the dependence on matched classes and the exclusion of all-zero or null rows is a validity limitation, not a circularity mechanism. No step in the paper fits the enumerated circular patterns, so the appropriate honest verdict is no significant circularity, with the observed inconsistency and matching limitation flagged as correctness risks rather than as circular reasoning.
Assumptions & free parameters
assumptions (3)
- domain assumption Understand's metric labels, such as LOC, CBO, LCOM, and DIT, are valid proxies for the corresponding software quality attributes.
- domain assumption Classes can be matched across versions so that paired Wilcoxon tests are meaningful, and cleaning or filtering nulls does not bias the comparison.
- ad hoc to paper A NaN result or a non-significant Wilcoxon test can be interpreted as stability rather than as missing data or a failed test.
Cite this review
Pith. "Pith review of Evolution analysis of software quality metrics in an open-source java project: A case study on TestNG." pith.science (2026). https://pith.science/paper/H4BBADM6
@misc{pith2026250522884,
author = {Pith},
title = {Pith review of: Evolution analysis of software quality metrics in an open-source java project: A case study on TestNG},
year = {2026},
howpublished = {\url{https://pith.science/paper/H4BBADM6}},
note = {Machine review of arXiv:2505.22884}
}
read the original abstract
Software quality is critical in modern software engineering, especially in large and evolving codebases. This study analyzes the evolution of software quality metrics in five successive versions of the open-source Java testing framework TestNG. Using the static analysis tool Understand, eleven key object-oriented metrics, including cyclomatic complexity, class coupling, and lines of code, were extracted for each version. Statistical and visual analyses reveal structural trends over time. The results indicate that TestNG has matured into a more stable and maintainable framework, reflecting ongoing development, refactoring, and architectural improvements. This study provides insights into design evolution and offers recommendations for maintaining code quality in similar projects.
Figures
Reference graph
Works this paper leans on
-
[1]
Shyam R. Chidamber and Chris F. Kemerer. A metrics suite for object-oriented design. IEEE Transactions on Software Engineering, 20(6):476–493, 1994
work page 1994
- [2]
-
[3]
Detection strategies: Metrics-based rules for detecting design flaws
Radu Marinescu. Detection strategies: Metrics-based rules for detecting design flaws. In Proceedings of the 20th IEEE International Conference on Software Maintenance (ICSM) , pages 350–359, 2004
work page 2004
-
[4]
Thomas Zimmermann, Nachiappan Nagappan, Harald Gall, Emanuel Giger, and Brendan Murphy. Predicting bugs from history. In Proceedings of the 27th international conference on Software engineering , pages 462–471, 2005. 9
work page 2005
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.