Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception

Anil Koyuncu; David Lo; Jacques Klein; Tegawend\'e F. Bissyand\'e; Wendk\^uuni C. Ou\'edraogo; Xin Zhou; Xueqi Dang; Yinghua Li

REVIEW 4 major objections 4 minor 29 references

Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read New metric CCTR tells structured test suites from fragmented ones, filling a blind spot in existing complexity measures.

desk verdict A test-specific complexity metric that is a good idea and a solid artifact, but the central claim that it reflects developer-perceived effort is unvalidated and the abstract overstates the evidence. read the letter →

arxiv 2506.06764 v2 pith:MR45APZ2 submitted 2025-06-07 cs.SE

Wendkûuni C. Ouédraogo , Yinghua Li , Xueqi Dang , Xin Zhou , Anil Koyuncu , Jacques Klein , David Lo , Tegawendé F. Bissyandé This is my paper

classification cs.SE

keywords cognitivecomplexityunittestingtestreadabilityCCTRLLM-generatedtestsEvoSuitesoftwaremetricscomprehension

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most complexity metrics used to judge automatically generated unit tests were built for production code, not test code, and the paper demonstrates the mismatch: SonarSource's Cognitive Complexity assigns zero to over 99% of LLM-generated test methods while also scoring simple nested dummy code higher than real test suites. The paper proposes CCTR, a test-aware metric that adds three test-specific signals—assertion density, mocking constructs, and annotation roles—to the traditional nesting-based score. Across 15,750 test suites generated by EvoSuite, GPT-4o, and Mistral Large over 350 Java classes, CCTR produces clearly separated scores for structured LLM tests and fragmented EvoSuite tests. The aim is to provide a readability-aware complexity score that can be used to evaluate, refactor, and improve generated tests.

What carries the argument

The central object is the CCTR formula, a weighted sum that treats four signals as equally important proxies for the mental effort of reading a test method: nesting complexity N, assertion count A, mocking count M, and annotation signaling T (with specialized annotations weighted double). This identity carries the argument because it converts the missing dimensions of test readability into a single comparable number, and the equal-weight choice (α = β = γ = δ = 1.0) is what keeps the metric lightweight and interpretable while producing the score separations reported in the evaluation.

What would settle it

Run a controlled study where developers rate the comprehension effort of test suites that are matched for CCTR score but differ in composition (e.g., assertion-heavy versus mock-heavy), or suites differing in CCTR but rated equally difficult. If the ratings do not track CCTR in either direction, the metric's claim to reflect developer-perceived effort fails. A concrete check: take two 12-method suites, one with 30 assertions and no mocks and one with 10 mocks and 5 assertions, both scoring the same CCTR total, and ask developers which is harder; any systematic preference breaks the equal-weight assumption.

Watch

Extended reading notes

Core claim

SonarSource's Cognitive Complexity, which penalizes nested control flow and logical operators, was designed for functional code and ignores the constructs that dominate unit tests. The paper claims that this makes it blind to the actual structure of test code: an LLM test suite with meaningful method names, assertions, and annotations scores 0, while a deeply nested dummy loop scores 12. CCTR is defined as CCTR = α·N + β·A + γ·M + δ·T, with all weights initially set to 1.0, where N is the original control-flow nesting complexity, A counts assertions and fail() statements, M counts mocking calls such as mock(), verify(), and when(), and T adds +1 per common annotation (@Test, @BeforeEach, @AfterEach) and +2 for @ParameterizedTest. On 15,750 suites, CCTR assigns mean scores of 26–30 to GPT-4o and Mistral tests and 39–59 to EvoSuite tests, separating generation styles that SonarSource's metric collapses. The paper concludes that CCTR better reflects test comprehension effort and structural variation in test code.

Load-bearing premise

The load-bearing premise is that counting nesting, assertions, mocks, and annotations with equal weight captures how hard developers actually find a test to read; if developers weight these factors differently, the metric's rankings may not match perceived effort.

Editorial extensions

If this is right

CCTR assigns nonzero scores to test suites that SonarSource's Cognitive Complexity rates as zero, so it can serve as a discriminating signal in automated review of generated tests.
Because CCTR scales to large batches, it enables comparison of test synthesis quality across tools and models where existing readability models or complexity metrics give little information.
The score differences between EvoSuite and LLM-generated suites quantify fragmentation and semantic density, giving a concrete target for test refactoring and generation prompts.
CCTR is intended to complement dynamic analyses: it covers structural and semantic effort, while coverage and fault-detection metrics cover behavioral effectiveness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The equal weights are an uncalibrated default; a developer study could find that one assertion and one mock call do not cost the same mental effort, in which case the ranking of generators would change even though the formula stays the same.
Because CCTR is purely syntactic, it can be gamed: merging assertions into a single helper call or deleting annotations would lower the score without necessarily improving readability, so a stress test under semantics-preserving transformations would show how stable the metric is.
The near-zero SonarSource scores for LLM tests may reflect that LLMs write flat, non-nested tests; CCTR's nonzero values then partly measure verbosity and annotation load, so the metric's 'effort' interpretation needs validation against actual comprehension times rather than structural intuition alone.
A natural next test is to check whether CCTR correlates with test usefulness, for example whether suites with very high CCTR (EvoSuite) also show lower fault-detection efficiency, linking readability scoring to test effectiveness.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: the paper defines CCTR, a simple additive metric for unit test complexity that counts nesting plus assertions, mocks, and annotations, reports it over 15,750 generated suites from EvoSuite, GPT-4o, and Mistral, and releases all data and scripts. That's a useful artifact, and the motivating observation is real: Sonar's Cognitive Complexity gives near-zero scores to most LLM-generated tests and fails to distinguish them from more fragmented EvoSuite tests. The metric itself is easy to understand and compute, and the authors are transparent about its limits in Section VI.

What is actually new: the combination of test-specific constructs into a single weighted sum is new relative to the literature they cite. It is a modest extension, not a deep framework, and they don't claim more than that. The release of the full dataset and evaluation scripts is a genuine plus; reproducibility is handled properly. The citation pattern is fine; the prior work they draw on is real, and the self-citations are to their own earlier test-generation studies, which is appropriate.

Now the soft spots. The core claim, in the abstract and in Section V-C, is that CCTR 'better reflects developer-perceived effort' and 'aligns with developer-centric principles.' As the paper's own Future Work admits, there is no human study. The weights are set to 1.0 without empirical derivation, and the authors call tuning future work. So the central validity claim is untested. I also think the empirical contrast in Table VI is partly a size artifact: EvoSuite suites have roughly two to three times the methods and LOC (Table III), and CCTR counts raw numbers of assertions, mocks, and annotations. Higher EvoSuite scores may just be longer classes, not more complex ones. The paper does not control for suite size or method count, and it does not compare against a baseline like LOC in predicting anything.

That said, I don't think these are fatal flaws in the sense of the metric being incoherent. They are missing validation and an overclaim relative to the evidence. If the authors weaken the language to 'CCTR captures structural variation in test constructs and is a candidate heuristic for test complexity,' the paper becomes a reasonable tool paper. As written, the abstract overpromises.

My take: send it to peer review, but expect a major-revision round. The artifact and the problem are real, and the measurement gap is worth attention. A serious referee can push for either a human study or a much more cautious conclusion. The data release makes that feasible.

Referee Report

4 major / 4 minor

Summary. The paper proposes CCTR, a test-aware cognitive complexity metric for unit tests, defined in Eq. (1) as a weighted sum of control-flow nesting, assertions, mocking constructs, and annotations. It evaluates CCTR on 15,750 test suites generated by EvoSuite, GPT-4o, and Mistral Large-1024 across Defects4J and SF110, comparing CCTR scores with Cyclomatic Complexity, SonarSource Cognitive Complexity, and Scalabrino readability scores. The paper claims that CCTR effectively discriminates between structured and fragmented test suites and "better reflects developer-perceived effort" (abstract, Section V-C). It also contributes a public dataset, prompts, and evaluation scripts.

Significance. A validated test-specific complexity metric would be a useful contribution, because existing metrics designed for functional code are known to assign near-zero scores to many generated tests. The paper has concrete strengths: it identifies a real gap, constructs a large and reproducible corpus (15,750 suites, public artifacts), and proposes a simple, interpretable metric that extends conventional cognitive complexity with test-specific constructs. However, the central claim that CCTR reflects developer-perceived effort is not supported by the evidence: the weights are asserted rather than derived or calibrated, no human ground truth is used, and the empirical comparison is descriptive and confounded with suite size. As it stands, the paper is best read as a proposal with exploratory observations, not as a validated metric.

major comments (4)

[Abstract and Section V-C] The claim that CCTR "better reflects developer-perceived effort" is not supported by the evidence presented. No human study or established readability ground truth is used; Section VI explicitly lists "a human study to validate the correlation between CCTR scores and developer-perceived test comprehensibility" as future work. The supporting evidence in Section V-C consists of score distributions (Table VI) and author-assigned perceived-complexity labels (Tables I and VII). These do not demonstrate that CCTR aligns with developer perception.
[Section V-B, Eq. (1)] The equal weights α = β = γ = δ = 1.0 are introduced without empirical justification, and Section V-B admits that weight tuning is future work. Since the metric's ranking of test suites depends directly on these weights, the result is not a measured property of test comprehension but an arbitrary design choice. A sensitivity analysis over weight combinations, or calibration against human judgments, is needed before the metric's scores can be interpreted as reflecting comprehension effort.
[Section V-C, Tables III and VI] The reported discrimination between EvoSuite and LLM-generated suites is confounded with suite size. According to Table III, EvoSuite suites have roughly 2–3× more LOC and methods per suite than LLM-generated suites, and CCTR sums raw counts of assertions, mocks, and annotations. Higher EvoSuite CCTR scores (Table VI) may therefore simply reflect longer, method-denser classes rather than the "fragmented structure" the paper attributes to them. The paper does not compare CCTR against LOC, method count, or other size baselines, nor does it normalize or control for size.
[Section III (A-B)] The treatment of non-compiling LLM-generated tests is not described precisely. Table III reports compilation rates of 100% for EvoSuite, 67.58% for GPT-4o, and 42.42% for Mistral Large-1024, but the paper does not state whether complexity and readability metrics were computed on all generated suites or only on those that compile. If non-compiling tests were included, PMD and CCTR scores may be unreliable; if they were excluded, the comparison is not based on the same population. This must be clarified and justified.

minor comments (4)

[Section I] There is a typo in the introduction: "congitive effort" should be "cognitive effort."
[Abstract and Section V] The metric is called "Test-Aware Cognitive Complexity" in the abstract but "Cognitive Complexity for Test Readability" in Section V; the naming should be made consistent.
[Table III] The Mistral-L rows for Defects4J and SF110 are identical (LOC 112.06, tokens 939.33, methods 16.20, compilability 42.42%). Please verify whether these values are correct or a copy-paste error.
[Tables I and VII] The "Perceived Complexity" labels are assigned by the authors and are used as if they were ground truth. They should be clearly presented as informal illustrations rather than empirical measurements.

Circularity Check

1 steps flagged · score 3.0 of 10

CCTR's formula is not circular, but its illustrative validation uses perceived-complexity labels built from the same features the metric counts; the claim of alignment with developer-perceived effort is deferred to future human study.

self definitional [Section II-D (Table I) and Section V-D (Table VII)]
"To better illustrate the disconnect between PMD's scoring and intuitive comprehension effort, we include a perceived complexity label based on structural and semantic clarity. ... These examples show that CCTR produces meaningful, differentiated scores at the class level, reflecting structural variation across test styles."

The perceived-complexity labels in Section II-D are assigned on the basis of the same structural and semantic properties (assertion clarity, naming, test structure) that Eq. (1) then hard-codes as N, A, M, and T. Section V-D presents the agreement between CCTR scores (12, 12, 35) and the author-assigned labels (Low, Medium, High) as evidence that CCTR reflects comprehension effort, but the agreement is by construction: the labels and the metric are both generated from the same feature set. This is an illustrative self-referential loop, not an independent validation against developer perception.

full rationale

CCTR's definition (Eq. 1) is an explicit, non-circular formula: a weighted sum of nesting complexity, assertion count, mock count, and annotation count, with weights initially set to 1.0. The weights are not fitted to the evaluation data, and the metric's values across generators are computed rather than predicted, so the formula itself does not reduce to its output. The self-citations [6] and [8] are used only to support the background claim that LLM-generated tests receive near-zero Cognitive Complexity scores; the paper independently reproduces that finding in Table V, so those citations are not load-bearing. The central weakness is evidential, not definitional: Section V-C asserts that higher EvoSuite scores 'support that CCTR better reflects test comprehension effort,' while Section VI concedes 'we aim to conduct a human study to validate the correlation between CCTR scores and developer-perceived test comprehensibility.' That is an overclaim about validity, not a circular derivation. The one genuine circular element is the example-based illustration: the perceived-complexity labels in Section II-D are derived from structural/semantic clarity, the metric is defined from the same features, and Section V-D then presents their agreement as meaningful. Because this loop is illustrative rather than the main quantitative evidence, and the main results are computed from an independent formula (not fitted to a subset and then 'predicted'), the overall circularity score is 3.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim relies on the assumption that the four chosen components and their equal weights capture developer-perceived test comprehension effort. No external validation against human judgment is provided; the paper explicitly defers that to future work. The weights alpha, beta, gamma, delta = 1.0 and the annotation bonus (+2 for @ParameterizedTest) are hand-chosen free parameters.

free parameters (5)

weight alpha for nesting component N = 1.0
Chosen by hand, not fitted to data. The paper states 'We adopt initial weights: alpha = beta = gamma = delta = 1.0' in Section V-B.
weight beta for assertion component A = 1.0
Chosen by hand, equal to all other weights, with no empirical calibration.
weight gamma for mocking component M = 1.0
Chosen by hand, equal to all other weights, with no empirical calibration.
weight delta for annotation component T = 1.0
Chosen by hand, equal to all other weights, with no empirical calibration.
annotation value for @ParameterizedTest = +2
The paper assigns +1 for common annotations and +2 for @ParameterizedTest. The extra +1 is a hand-chosen bonus without empirical support.

assumptions (3)

domain assumption The four components (nesting, assertions, mocking, annotations) adequately capture developer-perceived test comprehension effort.
The paper grounds this in Winkler et al. and Guerra et al., but the specific mapping to the metric components is the authors' design choice described in Section V-A.
ad hoc to paper Equal weights for the four components are appropriate for measuring perceived effort.
Section V-B sets all weights to 1.0 and explicitly states that tuning them is future work. This is an untested assumption on which the metric's validity depends.
domain assumption PMD's implementation of SonarSource Cognitive Complexity correctly computes the nesting component N for test code.
The paper uses PMD for existing metrics and says N 'follows the original SonarSource definition' (Section V-B), but does not question whether that definition behaves correctly on test-specific constructs.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception." pith.science (2026). https://pith.science/paper/MR45APZ2

@misc{pith2026250606764,
  author       = {Pith},
  title        = {Pith review of: Rethinking Cognitive Complexity for Unit Tests: Toward a Readability-Aware Metric Grounded in Developer Perception},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MR45APZ2}},
  note         = {Machine review of arXiv:2506.06764}
}

read the original abstract

Automatically generated unit tests-from search-based tools like EvoSuite or LLMs-vary significantly in structure and readability. Yet most evaluations rely on metrics like Cyclomatic Complexity and Cognitive Complexity, designed for functional code rather than test code. Recent studies have shown that SonarSource's Cognitive Complexity metric assigns near-zero scores to LLM-generated tests, yet its behavior on EvoSuite-generated tests and its applicability to test-specific code structures remain unexplored. We introduce CCTR, a Test-Aware Cognitive Complexity metric tailored for unit tests. CCTR integrates structural and semantic features like assertion density, annotation roles, and test composition patterns-dimensions ignored by traditional complexity models but critical for understanding test code. We evaluate 15,750 test suites generated by EvoSuite, GPT-4o, and Mistral Large-1024 across 350 classes from Defects4J and SF110. Results show CCTR effectively discriminates between structured and fragmented test suites, producing interpretable scores that better reflect developer-perceived effort. By bridging structural analysis and test readability, CCTR provides a foundation for more reliable evaluation and improvement of generated tests. We publicly release all data, prompts, and evaluation scripts to support replication.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

29 extracted references · 20 canonical work pages

[1]

Evosuite: automatic test suite generation for object-oriented software,

G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of soft- ware engineering, 2011, pp. 416–419

work page 2011
[2]

Using large language models to generate junit tests: An empirical study,

M. L. Siddiq, J. C. Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V . C. Lopes, “Using large language models to generate junit tests: An empirical study,” 2024

work page 2024
[3]

Chatgpt vs sbst: A comparative assessment of unit test suite generation,

Y . Tang, Z. Liu, Z. Zhou, and X. Luo, “Chatgpt vs sbst: A comparative assessment of unit test suite generation,” IEEE Transactions on Software Engineering, 2024

work page 2024
[4]

Chatunitest: A framework for llm-based test generation,

Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

2024
[5]

Software testing with large language models: Survey, landscape, and vision,

J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering , 2024

2024
[6]

Llms and prompting for unit test genera- tion: A large-scale evaluation,

W. C. Ouedraogo, K. Kabore, H. Tian, Y . Song, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyande, “Llms and prompting for unit test genera- tion: A large-scale evaluation,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 2464–2465

work page 2024
[7]

On the evaluation of large language models in unit test generation,

L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang et al., “On the evaluation of large language models in unit test generation,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , 2024, pp. 1607–1619

work page 2024
[8]

Large-scale, independent and comprehen- sive study of the power of llms for test case generation,

W. C. Ou ´edraogo, K. Kabor ´e, H. Tian, Y . Song, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyand ´e, “Large-scale, independent and comprehen- sive study of the power of llms for test case generation,” arXiv preprint arXiv:2407.00225, 2024

arXiv 2024

Show all 29 references

[9]

A complexity measure,

T. J. McCabe, “A complexity measure,” IEEE Transactions on software Engineering, no. 4, pp. 308–320, 1976

1976
[10]

Cognitive complexity: A new way of measuring under- standability,

G. A. Campbell, “Cognitive complexity: A new way of measuring under- standability,” https://www.sonarsource.com/docs/CognitiveComplexity. pdf, 2023, version 1.7

2023
[11]

Improving the readability of automatically generated tests using large language models,

M. Biagiola, G. Ghislotti, and P. Tonella, “Improving the readability of automatically generated tests using large language models,” arXiv preprint arXiv:2412.18843, 2024

2024 arXiv
[12]

Leveraging large language models for enhancing the understandability of generated unit tests,

A. Deljouyi, R. Koohestani, M. Izadi, and A. Zaidman, “Leveraging large language models for enhancing the understandability of generated unit tests,” arXiv preprint arXiv:2408.11710 , 2024

2024 arXiv
[13]

Investigating the readability of test code,

D. Winkler, P. Urbanke, and R. Ramler, “Investigating the readability of test code,” Empirical Software Engineering, vol. 29, no. 2, p. 53, 2024

2024
[14]

Modeling readability to improve unit tests,

E. Daka, J. Campos, G. Fraser, J. Dorn, and W. Weimer, “Modeling readability to improve unit tests,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering , 2015, pp. 107–118

2015
[15]

A comprehensive model for code readability,

S. Scalabrino, M. Linares-V ´asquez, R. Oliveto, and D. Poshyvanyk, “A comprehensive model for code readability,” Journal of Software: Evolution and Process , vol. 30, no. 6, p. e1958, 2018

2018
[16]

Reassessing java code readability models with a human- centered approach,

A. Sergeyuk, O. Lvova, S. Titov, A. Serova, F. Bagirov, E. Kirillova, and T. Bryksin, “Reassessing java code readability models with a human- centered approach,” in Proceedings of the 32nd IEEE/ACM International Conference on Program Comprehension , 2024, pp. 225–235

2024
[17]

Defects4j: A database of existing faults to enable controlled testing studies for java programs,

R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” in Pro- ceedings of the 2014 international symposium on software testing and analysis, 2014, pp. 437–440

2014
[18]

Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,

A. Panichella, F. M. Kifetew, and P. Tonella, “Automated test case generation as a many-objective optimisation problem with dynamic selection of the targets,” IEEE Transactions on Software Engineering , vol. 44, no. 2, pp. 122–158, 2017

2017
[19]

PMD An extensible cross-language static code analyzer

PMD, “PMD An extensible cross-language static code analyzer.” https: //pmd.github.io/, 2012, [Online; accessed 06-Jun-2024]

2012
[20]

Gpt-4o system card,

A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

2024 arXiv
[21]

A metric for software readability,

R. P. Buse and W. R. Weimer, “A metric for software readability,” in Proceedings of the 2008 international symposium on Software testing and analysis, 2008, pp. 121–130

2008
[22]

Learning a metric for code readability,

——, “Learning a metric for code readability,” IEEE Transactions on software engineering, vol. 36, no. 4, pp. 546–558, 2009

2009
[23]

A simpler model of software readability,

D. Posnett, A. Hindle, and P. Devanbu, “A simpler model of software readability,” in Proceedings of the 8th working conference on mining software repositories, 2011, pp. 73–82

2011
[24]

A general software readability model,

J. Dorn, “A general software readability model,” MCS Thesis available from (http://www. cs. virginia. edu/weimer/students/dorn-mcs-paper. pdf), vol. 5, pp. 11–14, 2012

2012
[25]

Towards using visual, semantic and structural features to improve code readability classification,

Q. Mi, Y . Hao, L. Ou, and W. Ma, “Towards using visual, semantic and structural features to improve code readability classification,” Journal of Systems and Software , vol. 193, p. 111454, 2022

2022
[26]

How do annotations affect java code readability?

E. Guerra, E. Gomes, J. Ferreira, I. Wiese, P. Lima, M. Gerosa, and P. Meirelles, “How do annotations affect java code readability?” Empirical Software Engineering , vol. 29, no. 3, p. 62, 2024

2024
[27]

Codebleu: a method for automatic evaluation of code synthesis,

S. Ren, D. Guo, S. Lu, L. Zhou, S. Liu, D. Tang, N. Sundaresan, M. Zhou, A. Blanco, and S. Ma, “Codebleu: a method for automatic evaluation of code synthesis,” arXiv preprint arXiv:2009.10297 , 2020

2009 arXiv
[28]

An empirical validation of cognitive complexity as a measure of source code understandability,

M. Mu ˜noz Bar ´on, M. Wyrich, and S. Wagner, “An empirical validation of cognitive complexity as a measure of source code understandability,” in Proceedings of the 14th ACM/IEEE international symposium on empirical software engineering and measurement (ESEM) , 2020, pp. 1–12

2020
[29]

Do we need improved code quality metrics?

T. Sharma and D. Spinellis, “Do we need improved code quality metrics?” arXiv preprint arXiv:2012.12324 , 2020

2012 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Evosuite: automatic test suite generation for object-oriented software,

G. Fraser and A. Arcuri, “Evosuite: automatic test suite generation for object-oriented software,” in Proceedings of the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of soft- ware engineering, 2011, pp. 416–419

work page 2011

[2] [2]

Using large language models to generate junit tests: An empirical study,

M. L. Siddiq, J. C. Santos, R. H. Tanvir, N. Ulfat, F. Al Rifat, and V . C. Lopes, “Using large language models to generate junit tests: An empirical study,” 2024

work page 2024

[3] [3]

Chatgpt vs sbst: A comparative assessment of unit test suite generation,

Y . Tang, Z. Liu, Z. Zhou, and X. Luo, “Chatgpt vs sbst: A comparative assessment of unit test suite generation,” IEEE Transactions on Software Engineering, 2024

work page 2024

[4] [4]

Chatunitest: A framework for llm-based test generation,

Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

2024

[5] [5]

Software testing with large language models: Survey, landscape, and vision,

J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering , 2024

2024

[6] [6]

Llms and prompting for unit test genera- tion: A large-scale evaluation,

W. C. Ouedraogo, K. Kabore, H. Tian, Y . Song, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyande, “Llms and prompting for unit test genera- tion: A large-scale evaluation,” in Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, 2024, pp. 2464–2465

work page 2024

[7] [7]

On the evaluation of large language models in unit test generation,

L. Yang, C. Yang, S. Gao, W. Wang, B. Wang, Q. Zhu, X. Chu, J. Zhou, G. Liang, Q. Wang et al., “On the evaluation of large language models in unit test generation,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering , 2024, pp. 1607–1619

work page 2024

[8] [8]

Large-scale, independent and comprehen- sive study of the power of llms for test case generation,

W. C. Ou ´edraogo, K. Kabor ´e, H. Tian, Y . Song, A. Koyuncu, J. Klein, D. Lo, and T. F. Bissyand ´e, “Large-scale, independent and comprehen- sive study of the power of llms for test case generation,” arXiv preprint arXiv:2407.00225, 2024

arXiv 2024