Pith. sign in

REVIEW 3 major objections 6 minor 102 references

The Ground Is Shifting: A Reflection on the Foundations of Software Measurement

T0 review · 3 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Software measurement's founding assumption\u2014that version-control traces are generated by human developers\u2014is now frequently violated by AI agents, so the field must detect violations and run a systematic AI-assisted replication…

desk verdict A sincere, well-grounded call to action on the erosion of the human-origin assumption in software measurement, but the 'frequently violate' claim outruns the evidence. read the letter →

arxiv 2608.03007 v1 pith:27QMVHXJ submitted 2026-08-04 cs.SE

Thomas Bock , Audris Mockus , Bogdan Vasilescu This is my paper · ORCID
classification cs.SE
keywords softwaremeasurementminingrepositorieshuman-originassumptionsquash-mergingAI-generatedcodereplicationprovenancemetadataempiricalengineering
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the dominant way of measuring software\u2014mining traces left in version-control and issue-tracking tools\u2014rests on an assumption that is silently dissolving: that the traces are generated by human developers building genuine systems. Changes in practice, especially squash-merging pull requests into single commits, and the arrival of AI agents that commit, review, and open pull requests mean contemporary traces no longer reliably reflect human activity. The paper contends that if this continues, classic findings about productivity, defects, and collaboration may not replicate on current data, and software measurement would lose relevance. It calls for detecting assumption violations, developing methods valid under new data-generation regimes, and a systematic AI-assisted replication program that re-runs key findings on modern data. If the paper is right, much of what empirical software engineering believes about how software is built would need to be re-examined.

What carries the argument

The carrying mechanism is the set of foundational assumptions listed in Table 1, with the human-origin assumption as the load-bearing one: a commit is an atomic unit of work, traces are generated by human developers, history is append-only, and author identity is unique and stable. Each assumption is paired with a modern violation\u2014squash-merging, AI-generated commits, force-pushing and rebasing, fragmented or shared identities\u2014and the paper uses these pairings to organize why analyses built on the assumptions fail. The worked example of squash-merging does the argumentative work of showing that a purely workflow-driven change can systematically distort commit-count productivity, churn, bug-introduction, collaboration-network, and temporal analyses; the AI-agent example extends the same logic to traces whose generative process is no longer human at all. These examples convert the abstract worry about shifting data generation into concrete, checkable threats to specific measurement constructs.

What would settle it

Measure the actual share of commits, pull requests, and reviews attributable to AI agents in a large public archive such as World of Code, using markers like bot accounts, self-admitted AI usage, and commit-message patterns; if the share is a tiny fraction of all traces, the claim that the human-origin assumption is 'frequently' violated is false. A second check would be to run replications of the three constructs in Table 3 on contemporary data: if commit-count productivity, code-churn defect prediction, and expertise-from-contributions findings all survive unchanged, the paper's central concern is not supported.

Watch

Extended reading notes

Core claim

The central claim is that the field's foundational 'human-origin assumption'\u2014that tool traces are produced by professional developers following established processes with quality gates\u2014is now frequently violated, and that the validity of software-measurement research therefore cannot be taken for granted. The paper supports this with two worked examples: squash-merging collapses multi-commit pull requests into a single commit, erasing development trajectory (with descriptive statistics showing 35\u201365% of merged PRs contain more than one commit in eight sampled projects and 39% of large active GitHub projects contain at least one squash-merged commit), and AI agents generate commits, messages, and reviews that look human but do not reflect human effort or cognition. From these examples, the paper concludes that current trace-based measurement would no longer be relevant without significant methodological change. The proposed remedy is a systematic, AI-assisted replication program that revisits classic findings on contemporary data, plus advocacy for provenance metadata so future commits can be labeled as human, AI-assisted, or AI-generated.

Load-bearing premise

The paper's urgency rests on the premise, asserted in Section 3.2 and not quantified, that a substantial and growing share of version-control traces is now produced by AI agents, so that the human-origin assumption is 'frequently' violated.

Editorial extensions

If this is right

  • Commit-count and churn-based productivity metrics become unreliable because squash-merging collapses many commits into one and AI agents can generate many commits quickly.
  • Bug-introduction analyses such as SZZ cannot localize defects inside squashed commits, because a bug and its fix may be subsumed in the same atomic change.
  • Classic findings on developer expertise, ownership, and collaboration may not replicate when expertise shifts to prompting and review or when agent commits dilute human authorship.
  • The field needs new measurement constructs that separate human effort from AI-assisted generation, and methods that remain valid when the human-origin assumption is relaxed.
  • Tool and platform owners should emit provenance metadata (author type, model, prompt, review decision) as first-class parts of the commit record.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own proposal contains a feedback loop it does not discuss: if AI assistants carry out the replications, and AI-generated traces are exactly what is undermining the old assumptions, the replications could inherit the same problems; the paper acknowledges in Section 3.3 that AI may misinterpret details and introduce biases, which means the remedy's reliability is not established.
  • A testable extension would be to estimate the current share of AI-generated commits in public archives and plot its growth; the paper's urgency scales with that share, which it does not measure.
  • The same assumption-violation logic applies to other trace sources the paper touches only lightly, such as issue trackers and code reviews, where AI agents now file issues and write review comments; a replication agenda could systematically check those traces too.
  • If squash-merging adoption continues to rise, historical comparisons across time become difficult not only because of AI but because the unit of observation itself changes; one could benchmark new metric definitions against fine-grained branch histories before squashing to quantify the distortion.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This reflection paper argues that empirical software engineering's reliance on tool traces (version control, issue trackers) rests on assumptions that are increasingly violated by modern development practices: commits as atomic units, human origin of traces, append-only history, and stable author identity. The authors provide new descriptive evidence on squash-merging: in eight curated GitHub projects, 35–65% of merged pull requests contain more than one commit (Table 2), and a World of Code analysis estimates that 39% of large public projects contain at least one squash-merged commit, with adoption rising from 18% in 2016 to 39% in 2025. They argue that AI agents now generate traces that 'frequently' violate the human-origin assumption, rendering current measurement methods irrelevant, and they propose a systematic AI-assisted replication program, new measurement constructs, and provenance metadata support from tool makers.

Significance. If substantiated, the paper identifies a genuinely important threat to the validity of a large body of software-engineering research and proposes a concrete response. The squash-merging statistics are a useful contribution, and the availability of scripts (Zenodo [11]) supports reproducibility. The paper is appropriately cautious in several places, acknowledging the detection heuristic's undercount and the need for human validation of AI-assisted replications. However, the paper's central urgency claim—that AI-generated traces are already prevalent enough to make current measurement irrelevant—is not backed by a quantitative estimate of AI-authored trace share, and the paper oscillates between binary and continuous conceptualizations of authorship. These gaps do not invalidate the thesis, but they require revision before the call for a replication program can be considered evidence-based.

major comments (3)
  1. [Section 3.2 and Abstract] The paper's central urgency claim—that traces 'frequently violate' the human-origin assumption and that without change 'the current way of software measurement via trace data would no longer be relevant'—rests on the estimated prevalence of AI-generated commits. The only support offered is citations to adoption studies [69, 82, 86], none of which quantifies the share of commits or pull requests in mined repositories that are authored by AI agents. The paper itself notes that human commits are 'hybrid artifacts' (Section 3.2), which makes the binary violation threshold undefined. Please provide a quantitative estimate (even a range) of the proportion of AI-authored or AI-assisted commits in datasets such as World of Code or GitHub Archive, or justify why such a measurement is currently impossible; otherwise the call for a systematic replication program is a reasonable precaution but not a demonstrated necessity.
  2. [Section 3.1, Table 2, World of Code analysis] The 39% project-level adoption rate counts a project as adopting squash-merging if it contains at least one commit whose title matches the pattern '(# PR number)'. This conflates occasional use with project-level adoption and is sensitive to the >1000-commit threshold; a project with one squashed PR among thousands of ordinary commits counts identically to a project with mandatory squash-merging. Moreover, the 8-project convenience sample in Table 2 shows that 35–65% of PRs contain more than one commit, but the paper does not report what fraction of commits in the population are lost or how the distribution of squash-merged commit share per project looks. Please report the distribution of squash-merged commits per project (e.g., percentage of merge commits that are squashed) and analyze the sensitivity of the adoption estimate to the detection heuristic and the project-size threshold, so that the claim that traces 'frequently' deviate from atomic-commit semantics is supported.
  3. [Section 3.2 and Table 1] The 'human origin' assumption is presented as binary while the paper acknowledges a spectrum (human- vs AI-assisted vs AI-generated). Since the violation is defined as 'human origin,' it is unclear what fraction of AI involvement constitutes a violation, and the proposed provenance metadata schema ('Was this commit human-authored, AI-generated, or AI-assisted?') does not resolve this. Please define an operational threshold or a graded measure of human authorship, and state how the proposed replication studies would handle the continuous nature of the construct.
minor comments (6)
  1. [Table 2] The table as typeset lacks clear column separators; for example, 'denoland/deno15 142' reads as part of the project name. Please format the table with explicit column spacing or a ruled table so that the number of PRs is distinguishable from the project name.
  2. [Section 3.1] The 18% in 2016 and 39% in 2025 project-level adoption numbers are not accompanied by a figure or table in the manuscript; the reader cannot verify this trend. If a plot is available on Zenodo, reference it directly in the text, or add a small figure showing the adoption curve.
  3. [Figure 2] Figure 2 is referenced but not visible in the provided manuscript text; please ensure the figure file is included and that the caption clearly identifies which commit messages demonstrate the difference between a traditional merge and a squash merge.
  4. [Table 3] The column header 'Productivity & effort estimation' uses an ampersand; use 'and' for formal text to avoid confusion with the LaTeX alignment character.
  5. [Section 3.2 and Abstract] The sentence 'Without a significant change, the current way of software measurement via trace data would no longer be relevant' appears verbatim in both the abstract and Section 3.2; consider rewording one of them to avoid exact duplication.
  6. [Introduction] The sentence 'This democratized empirical software engineering: Any project using these systems could analyze its own data and benchmark against historical states' would read more clearly with a semicolon or a period after 'software engineering'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the reflection's empirical premises rest on external studies and public archives, not on self-derived inputs.

full rationale

The paper is a position/reflection piece, not a chain of derivations with fitted parameters, so the circularity patterns do not apply. The central claim that AI agents 'frequently violate' the human-origin assumption is asserted in the abstract and Section 3.2, and it is supported by citations to external adoption studies [69, 82, 86]; it is not derived from the authors' own measurements. The squash-merging prevalence statistics in Section 3.1 are descriptive measurements computed from the public World of Code archive [51] and GitHub API data, with extraction scripts and plots publicly released on Zenodo [11]; even though one author is a co-creator of World of Code, this is a widely used public repository archive and the measurement is externally checkable, so the self-citation is not load-bearing in a circular sense. The proposed replication program is explicitly framed as a response to uncertainty ('We do not know how many classic findings survive under contemporary conditions'), and the paper acknowledges limitations of AI-assisted replication, including possible misinterpretation and bias. No equation is defined in terms of the conclusion, no fitted value is renamed as a prediction, and no uniqueness or ansatz is imported from the authors' prior work to force the central argument. The only weakness, an unquantified use of 'frequently' for AI-generated trace share, is a question of evidence strength rather than circularity and does not raise the circularity score.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper's argument rests on the premise that tool traces were the substrate of software measurement, that AI agents now produce a large enough share of traces to matter, and that squash-merging can be detected by a standardized commit-message pattern. The first is an explicit historical framing; the second is a cited premise not measured in this paper; the third is a detection heuristic with acknowledged undercounting. The only hand-chosen parameter is the 1,000-commit project size threshold used in the World of Code analysis.

free parameters (1)
  • Project size threshold = 1,000 commits
    Used to restrict the World of Code analysis to large projects; a hand-chosen cutoff (Sec. 3.1).
assumptions (3)
  • domain assumption Version-control traces are a valid proxy for software development activity when the human-origin assumption holds.
    The paper's entire framing depends on the idea that tool traces were and are the substrate of software measurement (Sec. 2).
  • domain assumption A growing fraction of repository traces is generated by AI agents, so the 'human origin' assumption is frequently violated.
    Central premise for urgency; supported only by citations to Robbes et al. [69] and others (Sec. 3.2).
  • domain assumption Squash-merged commits can be identified by the '(# PR number)' title pattern.
    Population-level estimates depend on this heuristic; the paper notes it undercounts but assumes it is indicative (Sec. 3.1).

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Ground Is Shifting: A Reflection on the Foundations of Software Measurement." pith.science (2026). https://pith.science/paper/27QMVHXJ

@misc{pith2026260803007,
  author       = {Pith},
  title        = {Pith review of: The Ground Is Shifting: A Reflection on the Foundations of Software Measurement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27QMVHXJ}},
  note         = {Machine review of arXiv:2608.03007}
}
abstract

For most of the past six decades, software measurement relied on labor-intensive manual collection of proprietary data, which hampered progress. The shift to repurposing traces from version control and related tools dramatically expanded data availability$\unicode{x2014}$especially with the rise of open-source software$\unicode{x2014}$but hinged on an often unstated assumption: that these tools are used by professional developers to build genuine software systems. However, as trace-generating tools, data types and scale, and empirical methods have all evolved, it has become clear that changes in data generation and analytical approaches affect many prior findings about software development, maintenance, and evolution. With AI agents now actively using these same tools, the resulting traces frequently violate the original assumption of human origin. To preserve the relevance of software measurement research, immediate action is needed: We must detect when foundational assumptions are violated in contemporary data and develop new methodologies that remain valid under changed circumstances. To this end, we propose a systematic AI-assisted replication program that revisits key findings using modern techniques, aiming for methods that yield consistent results on current data to keep software measurement meaningful.

Figures

Figures reproduced from arXiv: 2608.03007 by the authors.

Figure 1
Figure 1. The same pull request results in different histories: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

102 extracted references · 44 canonical work pages

  1. [11]

    T. Bock, A. Mockus, and B. Vasilescu. 2026.The adoption of squash-merging in GitHub projects – Scripts and descriptive statistics. doi:10.5281/zenodo.21727085

    work page doi:10.5281/zenodo.21727085 2026

  2. [1]

    Abreu, V

    R. Abreu, V. Murali, P. C. Rigby, C. Maddila, W. Sun, J. Ge, K. Chinniah, A. Mockus, M. Mehta, and N. Nagappan. 2025. Moving faster and reducing risk: Using LLMs in release deployment. InProc. Int. Conf. Softw. Eng.: Softw. Eng. in Practice (ICSE-SEIP). IEEE, 448–457. doi:10.1109/ICSE-SEIP66354.2025.00045

    arXiv 2025

  3. [2]

    Agroskin, E

    A. Agroskin, E. Lyulina, S. Titov, and V. Kovalenko. 2023. Constructing temporal networks of OSS programming language ecosystems. InInt. Conf. Softw. Analysis, Evolution, and Reengineering (SANER). IEEE, 663–667. doi:10.1109/SANER56733. 2023.00067

    arXiv 2023

  4. [3]

    A. J. Albrecht and J. E. Gaffney. 1983. Software function, source lines of code, and development effort prediction: A software science validation.IEEE Trans. Softw. Eng. (TSE)6 (1983), 639–648. doi:10.1109/TSE.1983.235271

    arXiv 1983

  5. [4]

    AlMarzouq, A

    M. AlMarzouq, A. AlZaidan, and J. AlDallal. 2020. Mining GitHub for research and education: Challenges and opportunities.Int. J. Web Inf. Syst. (IJWIS)16, 4 (2020), 451–473. doi:10.1108/IJWIS-03-2020-0016

    work page doi:10.1108/ijwis-03-2020-0016 2020

  6. [5]

    Ampatzoglou, S

    A. Ampatzoglou, S. Bibi, P. Avgeriou, M. Verbeek, and A. Chatzigeorgiou

  7. [6]

    V. R. Basili. 1992.Software modeling and measurement: The Goal/Question/Metric paradigm. Technical Report. https://www.cs.umd.edu/~basili/publications/ technical/T78.pdf

    1992

  8. [7]

    C. Bird, P. C. Rigby, E. T. Barr, D. J. Hamilton, D. M. German, and P. Devanbu

Show all 102 references
  1. [8]

    O. A. Blanthorn, C. M. Caine, and E. M. Navarro-López. 2019. Evolution of communities of software: Using tensor decompositions to compare software ecosystems.Appl. Netw. Sci.4, 1 (2019), 120. doi:10.1007/s41109-019-0193-5

    2019 doi

  2. [9]

    Bludau and A

    P. Bludau and A. Pretschner. 2022. PR-SZZ: How pull requests can support the tracing of defects in software repositories. InInt. Conf. Softw. Analysis, Evolution, and Reengineering (SANER). IEEE, 1–12. doi:10.1109/SANER53432.2022.00012

    2022

  3. [10]

    T. Bock, N. Alznauer, M. Joblin, and S. Apel. 2023. Automatic core-developer identification on GitHub: A validation study.ACM Trans. Softw. Eng. Methodol. (TOSEM)32, 6 (2023), 138. doi:10.1145/3593803

    2023 doi

  4. [12]

    B. W. Boehm. 1981.Software engineering economics. Prentice-Hall

    1981

  5. [13]

    Boldi, A

    P. Boldi, A. Pietri, S. Vigna, and S. Zacchiroli. 2020. Ultra-large-scale repository analysis via graph compression. InInt. Conf. Softw. Analysis, Evolution, and Reengineering (SANER). IEEE, 184–194. doi:10.1109/SANER48275.2020.9054827

    2020

  6. [14]

    Chatlatanagulchai, K

    W. Chatlatanagulchai, K. Thonglek, B. Reid, Y. Kashiwa, P. Leelaprute, A. Rung- sawang, B. Manaskasemsak, and H. Iida. 2025. On the use of agentic coding man- ifests: An empirical study of Claude Code. InInt. Conf. Product-Focused Softw. Pro- cess Improvement (PROFES). Springe...

    2025 doi

  7. [15]

    Cosentino, J

    V. Cosentino, J. L. Cánovas Izquierdo, and J. Cabot. 2017. A systematic mapping study of software development with GitHub.IEEE Access5 (2017), 7173–7192. doi:10.1109/ACCESS.2017.2682323

    2017

  8. [16]

    Cosentino, J

    V. Cosentino, J. Luis, and J. Cabot. 2016. Findings from GitHub: Methods, datasets and limitations. InProc. Int. Conf. Mining Softw. Repositories (MSR). ACM, 137–141. doi:10.1145/2901739.2901776

    2016

  9. [17]

    Dabić, R

    O. Dabić, R. Tufano, and G. Bavota. 2024. SEART data hub: Streamlining large- scale source code mining and pre-processing. InProc. Int. Conf. Softw. Maintenance and Evolution (ICSME). IEEE, 888–892. doi:10.1109/ICSME58944.2024.00097

    2024

  10. [18]

    F. G. de Oliveira Neto, R. Torkar, R. Feldt, L. Gren, C. A. Furia, and Z. Huang

  11. [19]

    T. Dey, A. Karnauch, and A. Mockus. 2021. Representation of developer expertise in open source software. InProc. Int. Conf. Softw. Eng. (ICSE). IEEE, 995–1007. doi:10.1109/ICSE43902.2021.00094

    2021

  12. [20]

    T. Dey, S. Mousavi, E. Ponce, T. Fry, B. Vasilescu, A. Filippova, and A. Mockus

  13. [21]

    Evolution of statistical analysis in empirical software engineering research: Current state and steps forward.J. Syst. Softw. (JSS)156 (2019), 246–267. doi:10. 1016/j.jss.2019.07.002

    2019

  14. [22]

    H. V. F. dos Santos, V. Costa, J. E. Montandon, and M. T. Valente. 2026. Decoding the configuration of AI coding agents: Insights from Claude Code projects. InInt. Workshop on Agentic Eng. (AGENT). ACM, 63–67. doi:10.1145/3786167.3788412

    2026

  15. [23]

    Dueñas, V

    S. Dueñas, V. Cosentino, J. M. Gonzalez-Barahona, A. del Castillo San Felix, D. Izquierdo-Cortazar, L. Cañas-Díaz, and A. Pérez García-Plaza. 2021. GrimoireLab: A toolset for software development analytics.PeerJ Comput. Sci.7, e601 (2021). doi:10.7717/peerj-cs.601

    2021 doi

  16. [24]

    U. K. Durrani, M. Akpinar, M. Fatih Adak, A. Talha Kabakus, M. Maruf Öztürk, and M. Saleh. 2024. A decade of progress: A systematic literature review on the integration of AI in software engineering phases and activities (2013–2023).IEEE Access12 (2024), 171185–171204. doi:10....

    2024

  17. [25]

    Di Penta and D

    M. Di Penta and D. A. Tamburri. 2017. Combining quantitative and qualitative studies in empirical software engineering research. InCompanion Volume ICSE. IEEE, 499–500. doi:10.1109/ICSE-C.2017.163

    2017 doi

  18. [26]

    Felderer and G

    M. Felderer and G. H. Travassos. 2020. The evolution of empirical methods in software engineering. InContemporary Empirical Methods in Software Engineering. Springer, 1–24. doi:10.1007/978-3-030-32489-6_1

    2020 doi

  19. [27]

    Feldt and A

    R. Feldt and A. Magazinius. 2010. Validity threats in empirical software engineer- ing research–An initial survey. InProc. Int. Conf. Softw. Eng. & Knowledge Eng. (SEKE). KSI Research, 374–379. https://www.cse.chalmers.se/~feldt/publications/ feldt_2010_validity_threats_in_ese...

    2010

  20. [28]

    Méndez Fernández and J.-H

    D. Méndez Fernández and J.-H. Passoth. 2019. Empirical software engineering: From discipline to interdiscipline.J. Syst. Softw. (JSS)148 (2019), 170–179. doi:10. 1016/j.jss.2018.11.019

    2019

  21. [29]

    T. Dybå, R. Prikladnicki, K. Rönkkö, C. B. Seaman, and J. Sillito. 2011. Qualitative research in software engineering.Empir. Softw. Eng. (EMSE)16, 4 (2011), 425–429. doi:10.1007/s10664-011-9163-y

    2011 doi

  22. [30]

    C. A. Furia, R. Torkar, and R. Feldt. 2022. Applying Bayesian analysis guidelines to empirical software engineering data: The case of programming languages and code quality.ACM Trans. Softw. Eng. Methodol. (TOSEM)31, 3 (2022), 40. doi:10.1145/3490953

    2022 doi

  23. [31]

    C. A. Furia, R. Torkar, and R. Feldt. 2023. Towards causal analysis of empirical software engineering data: The impact of programming languages on coding competitions.ACM Trans. Softw. Eng. Methodol. (TOSEM)33, 1 (2023), 13. doi:10. 1145/3611667

    2023

  24. [32]

    Gousios and D

    G. Gousios and D. Spinellis. 2012. GHTorrent: GitHub’s data from a firehose. In Proc. Int. Conf. Mining Softw. Repositories (MSR). IEEE, 12–21. doi:10.1109/MSR. 2012.6224294

    2012

  25. [33]

    S. L. France. 2024. Navigating software development in the ChatGPT and GitHub Copilot era.Bus. Horiz.67, 5 (2024), 649–661. doi:10.1016/j.bushor.2024.05.009

    2024 doi

  26. [34]

    Graf-Vlachy and S

    L. Graf-Vlachy and S. Wagner. 2024. Cleaning up confounding: Accounting for endogeneity using instrumental variables and two-stage models.ACM Trans. Softw. Eng. Methodol. (TOSEM)33, 8 (2024), 199. doi:10.1145/3674730

    2024 doi

  27. [35]

    T. L. Graves, A. F. Karr, J. S. Marron, and H. Siy. 2000. Predicting fault incidence using software change history.IEEE Trans. Softw. Eng. (TSE)26, 7 (2000), 653–661. doi:10.1109/32.859533

    2000 doi

  28. [36]

    H. He, R. He, H. Gu, and M. Zhou. 2021. A large-scale empirical study on Java library migrations: Prevalence, trends, and rationales. InProc. Europ. Softw. Eng. Conf. and the Int. Sympos. Foundations of Softw. Eng. (ESEC/FSE). ACM, 478–490. doi:10.1145/3468264.3468571

    2021

  29. [37]

    R. B. Grady and D. L. Caswell. 1987.Software metrics: Establishing a company-wide program. Prentice-Hall

    1987

  30. [38]

    Hoess, C

    N. Hoess, C. Paradis, R. Kazman, and W. Mauerer. 2025. Does the tool matter? Exploring some causes of threats to validity in mining software repositories. In Int. Conf. Softw. Analysis, Evolution, and Reengineering (SANER). IEEE, 645–656. doi:10.1109/SANER64311.2025.00067

    2025

  31. [39]

    Howison and K

    J. Howison and K. Crowston. 2004. The perils and pitfalls of mining Source- Forge. InProc. Int. Conf. Mining Softw. Repositories (MSR). IET, 7–11. doi:10.1049/ ic:20040467

    2004

  32. [40]

    S. Just, K. Herzig, J. Czerwonka, and B. Murphy. 2016. Switching to git: The good, the bad, and the ugly. InProc. Int. Sympos. Softw. Reliability Eng. (ISSRE). IEEE, 400–411. doi:10.1109/ISSRE.2016.38

    2016 doi

  33. [41]

    Herzig and A

    K. Herzig and A. Zeller. 2013. The impact of tangled code changes. InProc. Int. Conf. Mining Softw. Repositories (MSR). IEEE, 121–130. doi:10.1109/MSR.2013.6624018

    2013

  34. [43]

    Kalliamvakou, G

    E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian

  35. [44]

    Khalid, G

    A. Khalid, G. Badshah, N. Ayub, M. Shiraz, and M. Ghouse. 2023. Software defect prediction analysis using machine learning techniques.Sustainability15, 6 (2023),

    2023

  36. [45]

    S. M. Hasan Kabir, Md. Tanim Rahman, and Aunik Hasan Mridul. 2025. Software defect prediction using traditional machine learning and ensemble learning algo- rithms.Smart Wearable Technol.1 (2025), A9. doi:10.47852/bonviewswt52025645

    2025 doi

  37. [46]

    Kohler, D

    B. Kohler, D. Zollikofer, J. Einsiedler, A. Hoyle, and E. Ash. 2026. Read the paper, write the code: Agentic reproduction of social-science results.arXiv preprint arXiv:2604.21965(2026). doi:10.48550/arXiv.2604.21965

    2026 doi arXiv

  38. [47]

    P. Lago, P. Runeson, Q. Song, and R. Verdecchia. 2024. Threats to validity in software engineering—Hypocritical paper section or essential analysis?. InProc. ASE ’26, October 12–16, 2026, Munich, Germany Bock et al. Int. Sympos. Empir. Softw. Eng. and Measurement (ESEM). ACM, ...

    2024

  39. [48]

    N. Li, M. Shepperd, and Y. Guo. 2020. A systematic review of unsupervised learning techniques for software defect prediction.Inf. Softw. Technol. (IST)122 (2020), 106287. doi:10.1016/j.infsof.2020.106287

    2020

  40. [49]

    G. Lin, S. Wen, Q.-L. Han, J. Zhang, and Y. Xiang. 2020. Software vulnerability detection using deep neural networks: A survey.Proc. IEEE108, 10 (2020), 1825–1848. doi:10.1109/JPROC.2020.2993293

    2020

  41. [50]

    C. Liu, S. Chen, L. Fan, B. Chen, Y. Liu, and X. Peng. 2022. Demystifying the vulnerability propagation and its evolution via dependency trees in the npm ecosystem. InProc. Int. Conf. Softw. Eng. (ICSE). ACM, 672–684. doi:10.1145/ 3510003.3510142

    2022

  42. [51]

    Y. Ma, C. Bogart, S. Amreen, R. Zaretzki, and A. Mockus. 2019. World of Code: An infrastructure for mining the universe of open source VCS data. InProc. Int. Conf. Mining Softw. Repositories (MSR). IEEE, 143–154. doi:10.1109/MSR.2019.00031

    2019

  43. [52]

    B. A. Kitchenham, S. L. Pfleeger, and N. Fenton. 1995. Towards a framework for software measurement validation.IEEE Trans. Softw. Eng. (TSE)21, 12 (1995), 929–944. doi:10.1109/32.489070

    1995 doi

  44. [53]

    A. Mockus. 2007. Large-scale code reuse in open source software. InInt. Workshop Emerging Trends in FLOSS Research and Development (FLOSS). IEEE, 7. doi:10. 1109/FLOSS.2007.10

    2007

  45. [54]

    Mockus, R

    A. Mockus, R. T. Fielding, and J. D. Herbsleb. 2000. A case study of open source software development: The Apache server. InProc. Int. Conf. Softw. Eng. (ICSE). ACM, 263–272. doi:10.1145/337180.337209

    2000

  46. [55]

    Mockus, R

    A. Mockus, R. T. Fielding, and J. D. Herbsleb. 2002. Two case studies of open source software development: Apache and Mozilla.ACM Trans. Softw. Eng. Methodol. (TOSEM)11, 3 (2002), 309–346. doi:10.1145/567793.567795

    2002

  47. [56]

    Mockus and L

    A. Mockus and L. G. Votta. 2000. Identifying reasons for software changes using historic databases. InProc. Int. Conf. Softw. Maintenance (ICSM). IEEE, 120–130. doi:10.1109/ICSM.2000.883028

    2000

  48. [57]

    Mockus and D

    A. Mockus and D. M. Weiss. 2000. Predicting risk of software changes.Bell Labs Tech. J.5, 2 (2000), 169–180. doi:10.1002/bltj.2229

    2000 doi

  49. [58]

    Mujahid, D

    S. Mujahid, D. E. Costa, R. Abdalkareem, and E. Shihab. 2023. Where to go now? Finding alternatives for declining packages in the npm ecosystem. InProc. Int. Conf. Autom. Softw. Eng. (ASE). IEEE, 1628–1639. doi:10.1109/ASE56229.2023. 00119

    2023

  50. [59]

    T. J. McCabe. 1976. A complexity measure.IEEE Trans. Softw. Eng. (TSE)SE-2, 4 (1976), 308–320. doi:10.1109/TSE.1976.233837

    1976

  51. [61]

    G. P. Oliveira, A. F. C. Moura, N. A. Batista, M. A. Brandão, A. Hora, and M. M. Moro. 2023. How do developers collaborate? Investigating GitHub heterogeneous networks.Softw. Qual. J.31, 1 (2023), 211–241. doi:10.1007/s11219-022-09598-x

    2023 doi

  52. [62]

    L. F.M. Osorio, P. de A. dos Santos Neto, G. Avelino, and W. A.L. Lira. 2025. An evaluation of the impact of code generation tools on software development. In Simpósio Brasileiro de Sistemas de Informação (SBSI). SBC, 625–634. doi:10.5753/ sbsi.2025.246605

    2025

  53. [63]

    Pandey, P

    R. Pandey, P. Singh, R. Wei, and S. Shankar. 2024. Transforming software develop- ment: Evaluating the efficiency and challenges of GitHub Copilot in real-world projects.arXiv preprint arXiv:2406.17910(2024). doi:10.48550/arXiv.2406.17910

    2024 doi arXiv

  54. [64]

    S. L. Pfleeger and B. A. Kitchenham. 2025. Evidence-based software engineering guidelines revisited.IEEE Trans. Softw. Eng. (TSE)51, 3 (2025), 814–819. doi:10. 1109/TSE.2025.3526730

    2025

  55. [65]

    Pietri, D

    A. Pietri, D. Spinellis, and S. Zacchiroli. 2020. The Software Heritage graph dataset: Large-scale analysis of public software development history. InProc. Int. Conf. Mining Softw. Repositories (MSR). ACM, 1–5. doi:10.1145/3379597.3387510

    2020

  56. [66]

    J. D. Musa. 1979. Software reliability measurement.J. Syst. Softw. (JSS)1 (1979), 223–241. doi:10.1016/0164-1212(79)90023-2

    1979 doi

  57. [67]

    P. Rani, F. Petrulio, and A. Bacchelli. 2024. On refining the SZZ algorithm with bug discussion data.Empir. Softw. Eng. (EMSE)29, 5 (2024), 115. doi:10.1007/s10664- 024-10511-2

    2024 doi

  58. [68]

    Rapaport, L

    S. Rapaport, L. Pautet, S. Tardieu, and S. Zacchiroli. 2025. Altered histories in version control system repositories: Evidence from the trenches. InProc. Int. Conf. Autom. Softw. Eng. (ASE). IEEE, 2184–2195. doi:10.1109/ASE63991.2025.00181

    2025

  59. [69]

    Robbes, T

    R. Robbes, T. Matricon, T. Degueule, A. Hora, and S. Zacchiroli. 2026. Agentic much? Adoption of coding agents on GitHub.arXiv preprint arXiv:2601.18341 (2026). doi:10.48550/arXiv.2601.18341

    2026 doi arXiv

  60. [70]

    Robles, A

    G. Robles, A. Capiluppi, J. M. Gonzalez-Barahona, B. Lundell, and J. Gamalielsson

  61. [71]

    Ruohonen and Q

    J. Ruohonen and Q. Ramadan. 2025. Tracing vulnerability propagation across open source software ecosystems. InIFIP Int. Conf. Testing Software and Systems. Springer, 325–332. doi:10.1007/978-3-032-05188-2_21

    2025 doi

  62. [72]

    N. F. Schneidewind. 1992. Methodology for validating software metrics.IEEE Trans. Softw. Eng. (TSE)18, 5 (1992), 410–422. doi:10.1109/32.135774

    1992 doi

  63. [73]

    L. H. Putnam and W. Myers. 1991.Measures for excellence: Reliable software on time, within budget. Prentice-Hall

    1991

  64. [74]

    C. B. Seaman, R. Hoda, and R. Feldt. 2025. Qualitative research methods in software engineering: Past, present, and future.IEEE Trans. Softw. Eng. (TSE)51, 3 (2025), 783–788. doi:10.1109/TSE.2025.3538751

    2025

  65. [75]

    H. M. Shah, Q. Z. Syed, B. Shankaranarayanan, I. Palit, A. Singh, K. Raval, K. Savaliya, and T. Sharma. 2023. Mining and fusing productivity metrics with code quality information at scale. InProc. Int. Conf. Softw. Maintenance and Evolution (ICSME). IEEE, 563–567. doi:10.1109/...

    2023

  66. [76]

    Y. Shin, A. Meneely, L. Williams, and J. A. Osborne. 2011. Evaluating complexity, code churn, and developer activity metrics as indicators of software vulnerabili- ties.IEEE Trans. Softw. Eng. (TSE)37, 6 (2011), 772–787. doi:10.1109/TSE.2010.81

    2011 doi

  67. [77]

    D. I. K. Sjøberg and G. R. Bergersen. 2023. Construct validity in software engi- neering.IEEE Trans. Softw. Eng. (TSE)49, 3 (2023), 1374–1396. doi:10.1109/TSE. 2022.3176725

    2023

  68. [78]

    D. I. K. Sjøberg, T. Dybå, and M. Jørgensen. 2007. The future of empirical methods in software engineering research. InFuture of Softw. Eng. (FOSE). IEEE, 358–378. doi:10.1109/FOSE.2007.30

    2007 doi

  69. [79]

    Streit and L

    J. Streit and L. Feye. 2024. Benchmarking ongoing development output in real- life software projects. InInt. Conf. Product-Focused Softw. Process Improvement (PROFES). Springer, 19–34. doi:10.1007/978-3-031-78392-0_2

    2024 doi

  70. [80]

    Tahir, C

    T. Tahir, C. Gencel, G. Rasool, T. Umer, J. Rasheed, S. F. Yeo, and T. Cevik. 2023. Early software defects density prediction: Training the international software benchmarking cross projects data using supervised learning.IEEE Access11 (2023), 141965–141986. doi:10.1109/ACCESS...

    2023

  71. [81]

    C. B. Seaman. 1999. Qualitative methods in empirical studies of software engi- neering.IEEE Trans. Softw. Eng. (TSE)25, 4 (1999), 557–572. doi:10.1109/32.799955

    1999 doi

  72. [82]

    Treude and M.-A

    C. Treude and M.-A. Storey. 2025. Generative AI and empirical software engi- neering: A paradigm shift. InInt. Conf. AI-powered Softw. (AIware). IEEE, 233–239. doi:10.1109/AIware69974.2025.00033

    2025

  73. [83]

    Verdecchia, E

    R. Verdecchia, E. Engström, P. Lago, P. Runeson, and Q. Song. 2023. Threats to validity in software engineering research: A critical reflection.Inf. Softw. Technol. (IST)164 (2023), 107329. doi:10.1016/j.infsof.2023.107329

    2023

  74. [84]

    C. E. Walston and C. P. Felix. 1977. A method of programming measurement and estimation.IBM Syst. J.16, 1 (1977), 54–73. doi:10.1147/sj.161.0054

    1977 doi

  75. [85]

    Y. Wang, H. Le, A. Gotmare, N. Bui, J. Li, and S. Hoi. 2023. CodeT5+: Open code large language models for code understanding and generation. InProc. Conf. Empirical Methods in Natural Language Processing (EMNLP). ACL, 1069–1088. doi:10.18653/v1/2023.emnlp-main.68

    2023 doi

  76. [86]

    Watanabe, H

    M. Watanabe, H. Li, Y. Kashiwa, B. Reid, H. Iida, and A. E. Hassan. 2026. On the use of agentic coding: An empirical study of pull requests on GitHub.ACM Trans. Softw. Eng. Methodol. (TOSEM)(2026). Just accepted. doi:10.1145/3798166

    2026 doi

  77. [87]

    Weber, M

    T. Weber, M. Brandmaier, A. Schmidt, and S. Mayer. 2024. Significant productivity gains through programming with large language models.Proc. ACM Hum.Comput. Interact. (HCI)8, EICS (2024), 256. doi:10.1145/3661145

    2024 doi

  78. [88]

    E. J. Weyuker. 1988. Evaluating software complexity measures.IEEE Trans. Softw. Eng. (TSE)14, 9 (1988), 1357–1365. doi:10.1109/32.6178

    1988 doi

  79. [89]

    Terragni, A

    V. Terragni, A. Vella, P. Roop, and K. Blincoe. 2025. The future of AI-driven software engineering.ACM Trans. Softw. Eng. Methodol. (TOSEM)34, 5 (2025),

    2025

  80. [90]

    T. Xiao, Y. Fan, F. Calefato, C. Treude, R. G. Kula, H. Hata, and S. Baltes. 2026. Self-admitted GenAI usage in open-source software.IEEE Trans. Softw. Eng. (TSE) 52, 6 (2026), 1891–1910. doi:10.1109/TSE.2026.3681886

    2026

  81. [91]

    Xu and L

    Y. Xu and L. Y. Yang. 2026. Scaling reproducibility: An AI-assisted workflow for large-scale replication and reanalysis.arXiv preprint arXiv:2602.16733(2026). doi:10.48550/arXiv.2602.16733

    2026 doi arXiv

  82. [92]

    P. Xue, L. Wu, Z. Yu, Z. Jin, Z. Yang, X. Li, Z. Yang, and Y. Tan. 2024. Automated commit message generation with large language models: An empirical study and beyond.IEEE Trans. Softw. Eng. (TSE)50, 12 (2024), 3208–3224. doi:10.1109/TSE. 2024.3478317

    2024

  83. [93]

    Zerouali, T

    A. Zerouali, T. Mens, A. Decan, and C. De Roover. 2022. On the impact of security vulnerabilities in the npm and RubyGems dependency networks.Empir. Softw. Eng. (EMSE)27, 5 (2022), 107. doi:10.1007/s10664-022-10154-1

    2022 doi

  84. [94]

    Zhang, Z

    Y. Zhang, Z. Qiu, K.-J. Stol, W. Zhu, J. Zhu, Y. Tian, and H. Liu. 2024. Automatic commit message generation: A critical review and directions for future work. IEEE Trans. Softw. Eng. (TSE)50, 4 (2024), 816–835. doi:10.1109/TSE.2024.3364675

    2024

  85. [95]

    D. Zhou, Y. Wu, L. Xiao, Y. Cai, X. Peng, J. Fan, L. Huang, and H. Chen. 2019. Understanding evolutionary coupling by fine-grained co-change relationship analysis. InProc. Int. Conf. Program Comprehension (ICPC). IEEE, 271–282. doi:10. 1109/ICPC.2019.00046

    2019

  86. [96]

    Zimmermann, P

    T. Zimmermann, P. Weißgerber, S. Diehl, and A. Zeller. 2025. A retrospective on mining version histories to guide software changes.IEEE Trans. Softw. Eng. (TSE) 51, 3 (2025), 842–847. doi:10.1109/TSE.2025.3533559 Received 2026-05-13; accepted 2026-07-02

    2025

  87. [97]

    Y. Wu, Y. Wang, Y. Li, W. Tao, S. Yu, H. Yang, W. Jiang, and J. Li. 2025. An empirical study on commit message generation using LLMs via in-context learning. InProc. Int. Conf. Softw. Eng. (ICSE). IEEE, 553–565. doi:10.1109/ICSE55347.2025.00091

    2025

  88. [2009]

    The promises and perils of mining git. InProc. Int. Conf. Mining Softw. Repositories (MSR). IEEE, 1–10. doi:10.1109/MSR.2009.5069475

    2009

  89. [2014]

    The promises and perils of mining GitHub. InProc. Int. Conf. Mining Softw. Repositories (MSR). ACM, 92–101. doi:10.1145/2597073.2597074

  90. [2016]

    An in-depth study of the promises and perils of mining GitHub.Empir. Softw. Eng. (EMSE)21, 5 (2016), 2035–2071. doi:10.1007/s10664-015-9393-5

    2016 doi

  91. [2019]

    Identifying, categorizing and mitigating threats to validity in software engineering secondary studies.Inf. Softw. Technol. (IST)106 (2019), 201–230. doi:10.1016/j.infsof.2018.10.006

    2019 doi

  92. [2020]

    Detecting and characterizing bots that commit code. InProc. Int. Conf. Mining Softw. Repositories (MSR). ACM, 209–219. doi:10.1145/3379597.3387478

  93. [2022]

    Development effort estimation in free/open source software from activity in version control systems.Empir. Softw. Eng. (EMSE)27, 6 (2022), 135. doi:10. 1007/s10664-022-10166-x

    2022

  94. [5517]

    doi:10.3390/su15065517

    doi

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.