Pith. sign in

REVIEW 2 major objections 1 minor 57 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

Test suites leave 17.5% of documented expected behaviors untested even when line coverage and mutation scores are high.

2026-06-27 12:44 UTC pith:H64JKEQU

load-bearing objection The paper extracts expected behaviors from docs to measure a 17.5% untested gap and shows it is independent of coverage and mutation scores, but the extraction precision and mapping steps need more validation to support the number. the 2 major comments →

arxiv 2606.10417 v1 pith:H64JKEQU submitted 2026-06-09 cs.SE

Beyond Coverage and Kill Scores: Empirically Measuring Test Suite Behavioural Gaps

Partha Protim Paul , Reid Holmes This is my paper
classification cs.SE
keywords test suite adequacybehavioural coveragetest gapsmutation testingcode coverageautomated test generationJava libraries
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces an automated approach to pull expected method behaviors from documentation and code, then checks which ones existing tests actually cover. Across ten Java libraries it finds that 17.5% of these behaviors have no matching test at all. The same gaps appear in methods that already achieve high line coverage and high mutation kill scores. Automated test generators produce similar shortfalls. The results position behavioral coverage as a distinct dimension of test adequacy that structural metrics do not capture.

Core claim

The approach extracts expected method-level behaviours from natural language documentation and source code, maps them to existing test cases, and identifies gaps between expected and validated behaviours. Our empirical analysis conservatively estimates that 17.5% of detected expected behaviours remain entirely untested, which we term as the test suite's behavioural gap. Behavioural coverage acts as an independent dimension of test suite adequacy that can complement traditional structural metrics.

What carries the argument

An automated extraction method that pulls expected method-level behaviours from documentation and source code and maps them to test cases to reveal untested behaviours.

Load-bearing premise

Expected behaviours can be extracted from documentation and source code at 93.1% precision and mapped reliably to test cases.

What would settle it

A replication study on the same or similar libraries that uses different extraction rules and finds the share of untested behaviours to be near zero or strongly predicted by line coverage and mutation scores.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • 17.5% of extracted expected behaviours remain untested in the ten libraries studied.
  • EvoSuite and ASTER leave at least 20.6% and 27.1% of behaviours unvalidated respectively.
  • Most untested behaviours occur inside methods that already have high line coverage.
  • Over half of the gaps remain inside methods that have high mutation kill scores.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Teams could add documented-behaviour checks to existing test reports without replacing coverage tools.
  • The extraction technique might surface mismatches between documentation and implementation that developers would otherwise miss.
  • The same measurement could be applied to non-Java codebases or to evolving documentation over time.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 1 minor

Summary. The paper claims to introduce an automated proof-of-concept approach that extracts expected method-level behaviours from natural language documentation and source code, maps them to test cases, and identifies behavioural gaps in test suites. Evaluating on ten Java libraries (8,922 methods), it extracts 20,729 behaviours with 93.1% precision, conservatively estimating a 17.5% behavioural gap (untested expected behaviours). It further shows that automated test generators like EVOSUITE and ASTER leave similar or larger gaps (20.6%/27.1%), and that these gaps are not predicted by line coverage or mutation kill scores, positioning behavioural coverage as an independent adequacy dimension.

Significance. If the extraction precision and mapping accuracy hold, this work provides the first large-scale empirical evidence that traditional structural metrics miss a substantial portion of expected behaviours, suggesting a new complementary metric for test suite assessment. The scale of the study and the comparison to automated tools are strengths that could influence both research and practice in software testing.

major comments (2)
  1. [Abstract] Abstract: The 17.5% behavioural gap is computed directly from the 20,729 extracted behaviours whose extraction is reported at 93.1% precision; however, no details are supplied on the validation-set size, sampling procedure, or how false positives were adjudicated, leaving the reliability of the gap figure and the 'conservative' qualifier unsupported.
  2. [Evaluation section] Evaluation section: The mapping rule that decides whether an extracted behaviour is 'validated' by a test case is described only at high level; without an explicit definition or error-rate measurement for the mapping step, it is impossible to rule out systematic bias that would inflate or deflate the reported 17.5% gap.
minor comments (1)
  1. [Abstract] Abstract: The phrase 'conservatively estimates' is used without explaining which sources of under-counting (e.g., missed behaviours or extraction false negatives) are being treated conservatively.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments that identify areas where additional methodological transparency is needed. We address each point below and will revise the manuscript to supply the requested details.

read point-by-point responses

  1. Referee: [Abstract] Abstract: The 17.5% behavioural gap is computed directly from the 20,729 extracted behaviours whose extraction is reported at 93.1% precision; however, no details are supplied on the validation-set size, sampling procedure, or how false positives were adjudicated, leaving the reliability of the gap figure and the 'conservative' qualifier unsupported.

    Authors: We agree that the current manuscript does not report the validation-set size, sampling procedure, or adjudication criteria for the 93.1% precision figure. In the revised version we will add these details (including the number of behaviours manually inspected, how the sample was drawn, and the process used to classify false positives) so that readers can assess the reliability of the precision and the conservative behavioural-gap estimate. revision: yes

  2. Referee: [Evaluation section] Evaluation section: The mapping rule that decides whether an extracted behaviour is 'validated' by a test case is described only at high level; without an explicit definition or error-rate measurement for the mapping step, it is impossible to rule out systematic bias that would inflate or deflate the reported 17.5% gap.

    Authors: The referee is correct that the mapping rule is presented at a high level and lacks an explicit definition or measured error rate. We will revise the evaluation section to supply a formal definition of the validation mapping (including decision criteria and illustrative examples), report any available error-rate or agreement statistics, and discuss potential systematic biases and their possible effect on the 17.5% gap figure. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical extraction and direct counting

full rationale

The paper describes an empirical pipeline that extracts 20,729 method-level behaviours from documentation and source code (with a separately validated 93.1% precision on a sample), then maps those behaviours to existing test cases to compute the 17.5% untested fraction by direct enumeration. No equations, fitted parameters, self-citations, or ansatzes are used; the reported gap is simply the observed count of unmapped behaviours after the extraction step. The derivation chain is therefore self-contained and does not reduce any result to its own inputs by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The central claim rests on assumptions about documentation quality and extraction reliability rather than external benchmarks or formal proofs.

axioms (2)
  • domain assumption Natural language documentation and source code accurately describe the expected behaviors of methods.
    Invoked in the behavior extraction step that underpins all reported gaps.
  • ad hoc to paper 93.1% precision in behavior extraction is high enough to support the 17.5% untested estimate and comparisons to automated generators.
    Directly used to justify the validity of the 20,729 extracted behaviors.

pith-pipeline@v0.9.1-grok · 5804 in / 1400 out tokens · 28087 ms · 2026-06-27T12:44:13.729772+00:00 · methodology

0 comments
read the original abstract

Traditional test adequacy metrics measure a system's implementation, not whether it adheres to its expected behaviour. While developers rely heavily on code coverage and mutation testing to assess test suite quality, these metrics are fundamentally implementation-centric and cannot detect gaps between what the code is expected to do and what it actually does. Unfortunately, there has been no way to reliably detect these discrepancies; in this paper we introduce an automated proof-of-concept approach to investigate these gaps. The approach extracts expected method-level behaviours from natural language documentation and source code, maps them to existing test cases, and identifies gaps between expected and validated behaviours. We evaluate the approach across ten popular open-source Java libraries comprising 8,922 methods, extracting 20,729 behaviours with 93.1% precision. Our empirical analysis conservatively estimates that 17.5% of detected expected behaviours remain entirely untested, which we term as the test suite's behavioural gap. To determine if these gaps are merely an artifact of human-driven testing, we evaluate state-of-the-art automated test generators (EVOSUITE / ASTER), finding that they similarly fail to validate at least 20.6% / 27.1% of detected expected behaviours. We further demonstrate that behavioural gaps are not predicted by traditional structural metrics: the majority of untested behaviours occur in methods that already have high line coverage, and over half persist in methods with high mutation kill score. These results suggest behavioural coverage acts as an independent dimension of test suite adequacy that can complement traditional structural metrics.

Figures

Figures reproduced from arXiv: 2606.10417 by Partha Protim Paul, Reid Holmes.

Figure 1
Figure 1. Figure 1: Documentation, implementation, and developer-written [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Process BFINDER follows to identify any untested behaviours for a given documented method and the system’s existing developer-written test suite. source methods to the test cases that validate them. Sec￾ond, the behaviour extractor uses an LLM-based approach to identify the expected behaviours for any documented method. Finally, the method’s expected behaviours are mapped to the behaviours validated by the… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

57 extracted references · 9 canonical work pages · 2 internal anchors

  1. [1]

    Functional program testing,

    W. E. Howden, “Functional program testing,”Transactions on Software Engineering (TSE), vol. SE-6, pp. 162–169, 1978

    1978

  2. [2]

    Approaches to specification-based testing,

    D. J. Richardson, O. O’Malley, and C. Tittle, “Approaches to specification-based testing,” inProceedings of the Symposium on Testing, Analysis, and Verification (TAV3), 1989, pp. 86–96

    1989

  3. [3]

    G. J. Myers,The Art of Software Testing. John Wiley & Sons, 1979

    1979

  4. [4]

    Doc2OracLL: Investigating the impact of documentation on LLM-based test oracle generation,

    S. B. Hossain, R. Taylor, and M. Dwyer, “Doc2OracLL: Investigating the impact of documentation on LLM-based test oracle generation,”

  5. [5]

    Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generation

    [Online]. Available: https://arxiv.org/abs/2412.09360

    work page Pith review arXiv

  6. [6]

    Behaviourally adequate software testing,

    G. Fraser and N. Walkinshaw, “Behaviourally adequate software testing,” inProceedings International Conference on Software Testing, Verifica- tion and Validation (ICST), 2012, pp. 300–309

    2012

  7. [7]

    Reliability of the path analysis testing strategy,

    W. Howden, “Reliability of the path analysis testing strategy,”Trans- actions on Software Engineering (TSE), vol. SE-2, no. 3, pp. 208–215, 1976

    1976

  8. [8]

    The design of a prototype mutation system for program testing,

    T. A. Budd, R. J. Lipton, F. G. Sayward, and R. A. DeMillo, “The design of a prototype mutation system for program testing,” inProceedings of the National Computer Conference, 1978, pp. 623–627

    1978

  9. [9]

    Software unit test coverage and adequacy,

    H. Zhu, P. A. V . Hall, and J. H. R. May, “Software unit test coverage and adequacy,”ACM Computing Surveys (CSUR), vol. 29, no. 4, pp. 366–427, 1997

    1997

  10. [10]

    Mutation testing advances: an analysis and survey,

    M. Papadakis, M. Kintis, J. Zhang, Y . Jia, Y . Le Traon, and M. Harman, “Mutation testing advances: an analysis and survey,” inAdvances in computers. Elsevier, 2019, vol. 112, pp. 275–378

    2019

  11. [11]

    Code coverage for suite eval- uation by developers,

    R. Gopinath, C. Jensen, and A. Groce, “Code coverage for suite eval- uation by developers,” inProceedings of the International Conference on Software Engineering (ICSE), 2014, pp. 72–82

    2014

  12. [12]

    Are mutants a valid substitute for real faults in software testing?

    R. Just, D. Jalali, L. Inozemtseva, M. D. Ernst, R. Holmes, and G. Fraser, “Are mutants a valid substitute for real faults in software testing?” in Proceedings of the International Symposium on Foundations of Software Engineering (ESEC/FSE), 2014, pp. 654–665

    2014

  13. [13]

    Functional testing, structural testing and code reading: What fault type do they each detect?

    N. Juristo and S. Vegas, “Functional testing, structural testing and code reading: What fault type do they each detect?” inEmpirical Methods and Studies in Software Engineering. Springer Berlin Heidelberg, 2003, pp. 208–232

    2003

  14. [14]

    How well do professional developers test with code coverage visualizations? an empirical study,

    J. Lawrance, S. Clarke, M. Burnett, and G. Rothermel, “How well do professional developers test with code coverage visualizations? an empirical study,” inSymposium on Visual Languages and Human- Centric Computing (VL/HCC), pp. 53–60

  15. [15]

    Assertions are strongly correlated with test suite effectiveness,

    Y . Zhang and A. Mesbah, “Assertions are strongly correlated with test suite effectiveness,” inProceedings of the Joint Meeting on Foundations of Software Engineering (ESEC/FSE), 2015, pp. 214–224

    2015

  16. [16]

    Does mutation testing improve testing practices?

    G. Petrovi ´c, M. Ivankovi ´c, G. Fraser, and R. Just, “Does mutation testing improve testing practices?” inProceedings of the International Conference on Software Engineering (ICSE), 2021, pp. 910–921

    2021

  17. [17]

    Systematic mistake analysis of digital computer programs,

    J. C. Miller and C. J. Maloney, “Systematic mistake analysis of digital computer programs,”Communications of the ACM (CACM), vol. 6, no. 2, pp. 58–63, 1963

    1963

  18. [18]

    Toward a theory of test data selection,

    J. B. Goodenough and S. L. Gerhart, “Toward a theory of test data selection,”ACM SIGPLAN Notices, vol. 10, no. 6, pp. 493–510, Apr. 1975

    1975

  19. [19]

    Foundations of software testing: dependability theory,

    D. Hamlet, “Foundations of software testing: dependability theory,” SIGSOFT Software Engineering Notes (SEN), vol. 19, no. 5, pp. 128– 139, Dec. 1994

    1994

  20. [20]

    Coverage is not strongly correlated with test suite effectiveness,

    L. Inozemtseva and R. Holmes, “Coverage is not strongly correlated with test suite effectiveness,” inProceedings International Conference on Software Engineering (ICSE), 2014, pp. 435–445

    2014

  21. [21]

    The risks of coverage-directed test case generation,

    G. Gay, M. Staats, M. Whalen, and M. P. E. Heimdahl, “The risks of coverage-directed test case generation,”Transactions on Software Engineering (TSE), vol. 41, no. 8, pp. 803–819, 2015

    2015

  22. [22]

    Metamon: Finding inconsistencies be- tween program documentation and behavior using metamorphic llm queries,

    H. Lee, G. An, and S. Yoo, “Metamon: Finding inconsistencies be- tween program documentation and behavior using metamorphic llm queries,” inInternational Workshop on Large Language Models for Code (LLM4Code), 2025, pp. 120–127

    2025

  23. [23]

    Can large language models transform natural language intent into formal method postconditions?

    M. Endres, S. Fakhoury, S. Chakraborty, and S. K. Lahiri, “Can large language models transform natural language intent into formal method postconditions?”Proceedings of the ACM on Software Engineering, vol. 1, pp. 1889–1912, 2023

    1912

  24. [24]

    EvoSuite: Automatic test suite generation for object-oriented software,

    G. Fraser and A. Arcuri, “EvoSuite: Automatic test suite generation for object-oriented software,” inProceedings of the European Conference on Foundations of Software Engineering (ESEC), 2011, pp. 416–419

    2011

  25. [25]

    Aster: Natural and multi-language unit test generation with llms,

    R. Pan, M. Kim, R. Krishna, R. Pavuluri, and S. Sinha, “Aster: Natural and multi-language unit test generation with llms,”Proceedings of the International Conference on Software Engineering: Software Engineer- ing in Practice (ICSE-SEIP), pp. 413–424, 2024

    2024

  26. [26]

    Is mutation an appropriate tool for testing experiments?

    J. H. Andrews, L. C. Briand, and Y . Labiche, “Is mutation an appropriate tool for testing experiments?” inProceedings of the International Conference on Software Engineering (ICSE), 2005, pp. 402–411

    2005

  27. [27]

    Checked coverage: an indicator for oracle quality,

    D. Schuler and A. Zeller, “Checked coverage: an indicator for oracle quality,”Software Testing, Verification and Reliability, vol. 23, no. 7, pp. 531–551, 2013. [Online]. Available: https://onlinelibrary.wiley.com/ doi/abs/10.1002/stvr.1497

    work page doi:10.1002/stvr.1497 2013

  28. [28]

    A compre- hensive study of pseudo-tested methods,

    O. Vera P ´erez, B. Danglot, M. Monperrus, and B. Baudry, “A compre- hensive study of pseudo-tested methods,”Empirical Software Engineer- ing, vol. 24, 06 2019

    2019

  29. [29]

    Where tests fall short: empirically analyzing oracle gaps in covered code,

    M. Maton, G. M. Kapfhammer, and P. McMinn, “Where tests fall short: empirically analyzing oracle gaps in covered code,” inProceedings of the International Symposium on Empirical Software Engineering and Measurement (ESEM 2025). Institute of Electrical and Electronics Engineers (IEEE), 2025

    2025

  30. [30]

    Automatically gen- erating test cases for safety-critical software via symbolic execution,

    E. Kurian, D. Briola, P. Braione, and G. Denaro, “Automatically gen- erating test cases for safety-critical software via symbolic execution,” Journal of Systems and Software, vol. 199, p. 111629, 2023

    2023

  31. [31]

    Call me maybe: Using NLP to automatically generate unit test cases respecting temporal con- straints,

    A. Blasi, A. Gorla, M. D. Ernst, and M. Pezz `e, “Call me maybe: Using NLP to automatically generate unit test cases respecting temporal con- straints,” inProceedings of the International Conference on Automated Software Engineering (ASE), 2022, pp. 1–11

    2022

  32. [32]

    Nanofuzz: A usable tool for automatic test generation,

    M. C. Davis, S. Choi, S. Estep, B. A. Myers, and J. Sunshine, “Nanofuzz: A usable tool for automatic test generation,” ser. ESEC/FSE

  33. [33]

    1114–1126

    New York, NY , USA: Association for Computing Machinery, 2023, p. 1114–1126. [Online]. Available: https://doi.org/10.1145/ 3611643.3616327

    work page arXiv 2023

  34. [34]

    Feedback-directed random test generation,

    C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball, “Feedback-directed random test generation,” inProceedings of the International Conference on Software Engineering (ICSE), 2007, pp. 75–84

    2007

  35. [35]

    Chatunitest: A framework for llm-based test generation,

    Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” inCompanion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576

    2024

  36. [36]

    Automatic generation of oracles for exceptional behaviors,

    A. Goffi, A. Gorla, M. D. Ernst, and M. Pezz `e, “Automatic generation of oracles for exceptional behaviors,” inProceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2016, pp. 213– 224

    2016

  37. [37]

    Translating code comments to procedure spec- ifications,

    A. Blasi, A. Goffi, K. Kuznetsov, A. Gorla, M. D. Ernst, M. Pezz `e, and S. D. Castellanos, “Translating code comments to procedure spec- ifications,” inProceedings of the International Symposium on Software Testing and Analysis (ISSTA), 2018, pp. 242–253

    2018

  38. [38]

    Automated test generation from program documentation encoded in code comments,

    G. Denaro and L. Guglielmo, “Automated test generation from program documentation encoded in code comments,” 2025. [Online]. Available: https://arxiv.org/abs/2504.21161

    work page arXiv 2025

  39. [39]

    A framework for creating non-regressive test cases via branch consistency analysis driven by descriptions,

    Y . Zhang, P. Xue, Z. Yang, X. Ren, X. Li, L. Wu, J. Zhao, and X. Yu, “A framework for creating non-regressive test cases via branch consistency analysis driven by descriptions,” 2025. [Online]. Available: https://arxiv.org/abs/2506.07486

    work page arXiv 2025

  40. [40]

    Docprism: Local categorization and external filtering to identify relevant code-documentation inconsistencies,

    X. Xu, Z. Wahab, R. Holmes, and C. Lemieux, “Docprism: Local categorization and external filtering to identify relevant code-documentation inconsistencies,” 2025. [Online]. Available: https: //arxiv.org/abs/2511.00215

    work page arXiv 2025

  41. [41]

    Natural language generation and understanding of big code for ai-assisted programming: A review,

    M.-F. Wong, S. Guo, C.-N. Hang, S.-W. Ho, and C.-W. Tan, “Natural language generation and understanding of big code for ai-assisted programming: A review,”Entropy, vol. 25, no. 6, p. 888, Jun. 2023

    2023

  42. [42]

    Communicating study design trade-offs in software engineering,

    M. P. Robillard, D. M. Arya, N. A. Ernst, J. L. C. Guo, M. Lamothe, M. Nassif, N. Novielli, A. Serebrenik, I. Steinmacher, and K.-J. Stol, “Communicating study design trade-offs in software engineering,”ACM Transactions on Software Engineering Methodology (TOSEM), vol. 33, no. 5, Jun. 2024

    2024

  43. [43]

    Qwen3 Technical Report

    Q. Team, “Qwen3 technical report,” 2025. [Online]. Available: https://arxiv.org/abs/2505.09388

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  44. [44]

    Defects4J: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4J: A database of existing faults to enable controlled testing studies for java programs,” inProceed- ings of the International Symposium on Software Testing and Analysis (ISSTA), 2014, pp. 437–440

    2014

  45. [45]

    JaCoCo Java Code Coverage Li- brary,

    EclEmma and JaCoCo Contributors, “JaCoCo Java Code Coverage Li- brary,” https://www.jacoco.org/jacoco/trunk/doc/maven.html, 2026, ac- cessed: 2025-12-25

    2026

  46. [46]

    PIT: Mutation Testing for Java,

    PIT Contributors, “PIT: Mutation Testing for Java,” https://pitest.org/, 2026, accessed: 2025-12-22

    2026

  47. [47]

    Judging LLM-as-a-judge with MT-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. Xinget al., “Judging LLM-as-a-judge with MT-bench and chatbot arena,”Advances in neural information processing systems, vol. 36, pp. 46 595–46 623, 2023

    2023

  48. [48]

    Ministral 3

    A. H. Liu, K. Khandelwal, S. Subramanian, V . Jouault, A. Rastogi et al., “Ministral 3,” 2026. [Online]. Available: https://arxiv.org/abs/ 2601.08584

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  49. [49]

    Can llms replace manual annotation of software engineering artifacts?

    T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can llms replace manual annotation of software engineering artifacts?” inProceedings of the International Conference on Mining Software Repositories (MSR), 2025, pp. 526–538

    2025

  50. [50]

    Llm-as-a-judge for software engineering: Literature review, vision, and the road ahead,

    J. He, J. Shi, T. Y . Zhuo, C. Treude, J. Sun, Z. Xing, X. Du, and D. Lo, “Llm-as-a-judge for software engineering: Literature review, vision, and the road ahead,”Transactions on Software Engineering and Methodology (TOSEM), Feb. 2026

    2026

  51. [51]

    An llm-as-judge metric for bridging the gap with human evaluation in se tasks,

    X. Zhou, K. Kim, T. Zhang, M. Weyssow, L. F. Gomes, G. Yang, K. Liu, X. Xia, and D. Lo, “An llm-as-judge metric for bridging the gap with human evaluation in se tasks,”arXiv preprint arXiv:2505.20854, 2025

    work page arXiv 2025

  52. [52]

    Evaluation of sampling and cross-validation tuning strategies for regional-scale machine learning classification,

    C. A. Ramezan, T. A. Warner, and A. E. Maxwell, “Evaluation of sampling and cross-validation tuning strategies for regional-scale machine learning classification,”Remote Sensing, vol. 11, no. 2, 2019. [Online]. Available: https://www.mdpi.com/2072-4292/11/2/185

    2019

  53. [53]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,”Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  54. [54]

    A tutorial on sample size calculation for inter-rater and intra-rater agreement studies,

    F. Madadizadeh and S. Bahariniya, “A tutorial on sample size calculation for inter-rater and intra-rater agreement studies,”Indian Journal of Psychological Medicine, Feb. 2026

    2026

  55. [55]

    R. L. Scheaffer, W. Mendenhall, III, R. L. Ott, and K. G. Gerow, Elementary Survey Sampling, 7th ed. Brooks/Cole, 2012, ch. 5.6

    2012

  56. [56]

    Mutation testing advances: An analysis and survey,

    M. Papadakis, M. Kintis, J. Zhang, Y . Jia, Y . L. Traon, and M. Harman, “Mutation testing advances: An analysis and survey,” ser. Advances in Computers, A. M. Memon, Ed. Elsevier, 2019, vol. 112, pp. 275–378

    2019

  57. [57]

    Spearman,The proof and measurement of association between two things.Appleton-Century-Crofts, 1961

    C. Spearman,The proof and measurement of association between two things.Appleton-Century-Crofts, 1961

    1961