Pith. sign in

REVIEW 4 major objections 4 minor 93 references

Retrospective: Data Mining Static Code Attributes to Learn Defect Predictors

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This retrospective claims that publishing a baseline result with its data and scripts is enough to grow a research community, and uses a 2007 defect-prediction paper as the case study.

desk verdict A candid first-person history of the PROMISE community, with a plausible but untested causal claim; review it as a perspective piece, not a research paper. read the letter →

arxiv 2501.15662 v1 pith:WAQQDLK5 submitted 2025-01-26 cs.SE

Tim Menzies This is my paper · ORCID
classification cs.SE
keywords defectpredictionreproducibleresearchdatasharingsoftwareengineeringstaticcodemetricsopensciencebaselineresultdatasetreuse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This retrospective argues that a research community can be created at will: publish a solid baseline result together with the data and scripts needed to recreate it, and other researchers will take up the result, extend it, and build a field around it. The case study is a 2007 software-defect-prediction paper that shared exactly that, and whose shared artifacts were later used in a fifth of leading software-engineering papers. The essay also warns that the same shared resource can outlive its usefulness, turning from a gold standard into a constraint on progress. A sympathetic reader should take away a recipe for building open research communities and a caution about dataset stagnation.

What carries the argument

The mechanism is the publish-the-baseline loop: an author produces a first credible result, releases the dataset and all scripts needed to reproduce it, and invites others to beat it. The named object in the case study is the shared artifact repository and its companion conference, which supplied hundreds of software-engineering datasets and the scripts that made re-analysis cheap. This loop does the work by lowering the cost of entry for other researchers, turning a single paper into a standard benchmark, and creating a visible target that the field can collectively improve on.

What would settle it

One could compare two groups of papers on the same new topic, one published with full data and scripts and one without, matched for venue and author promotion, then count citations, replications, and derivative studies after the same time window; if shared-artifact papers do not grow faster, the central causal claim fails.

Watch

Extended reading notes

Core claim

The central discovery claimed is that open artifact sharing—not just the intellectual contribution of the model—was the load-bearing ingredient that turned a single defect-prediction baseline into a sustained, reproducible research movement. According to the retrospective, the 2007 paper's decision to release its data-mining scripts and datasets allowed hundreds of later studies to compare against, reuse, and improve the baseline; at its peak in 2016 the paper was software engineering's most cited paper per month, and by 2018 a fifth of leading journal articles used data introduced by that line of work. The paper further asserts that the same mechanism can be repeated for any research topic: a community forms when a baseline plus reproduction scripts are published. It also records a set of empirical observations about that body of work, including that different projects favour different metrics, static code attributes matter only in combination, much of the collected data can be discarded without hurting predictions, data-quality flaws matter less than expected, weak learners can still rank solutions usefully, and many supposedly hard problems yield to simpler methods.

Load-bearing premise

The whole story assumes that the shared data and scripts, rather than being first, the venue's prestige, or the author's promotion, caused the paper's popularity and the community's growth; no comparison group or counterfactual is tested.

Editorial extensions

If this is right

  • Publishing a reproducible baseline can seed a new research community on any topic, not just defect prediction.
  • A shared dataset quickly becomes the standard benchmark, which accelerates early growth but later risks locking a field into outdated data.
  • Defect prediction built on static code attributes is viable and has moved into industry, where surveys show most practitioners willing to adopt it and case studies report reduced inspection effort.
  • Methodologically, simple comparisons and cheap baselines should benchmark sophisticated methods; in this body of work simpler approaches repeatedly matched or beat complex ones.
  • Data quality, within limits, is less important than the predictive signal: injecting known quality issues into datasets did not degrade learned-model performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the causal recipe is right, research fields could deliberately seed new topics by publishing baselines; a testable extension is for new fields to try the same recipe and measure citation or reuse growth against comparable papers without shared artifacts.
  • The four-phase trajectory of shared repositories implies that such resources need governance—sunsetting, refresh, or quality re-certification—something the paper only gestures at.
  • The transfer-learning observation, where a model learned on one industry's data predicts failures in an unrelated system, hints that software metrics live on a low-dimensional structure; if so, data-efficient methods should keep outperforming brute-force data collection on tabular software-engineering tasks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This arXiv note is a retrospective on Menzies, Greenwald, and Frank's 2007 IEEE TSE paper on data-mining static code attributes for defect prediction. The author, also the originator of the PROMISE project that hosted the data and scripts, argues that the 2007 paper's open sharing of a baseline result, data, and scripts catalyzed a research community and that industry can deliberately create such communities by publishing baselines with reproducible artifacts. The note reviews the paper's technical claims, reports progress in defect prediction, proposes a four-phase model of shared-data lifecycles (from resistance to stagnation), lists nine 'Menzies's Laws' about software-engineering data mining, and closes with future directions, including changes to his own editorial policy at the Automated Software Engineering journal. The manuscript is written as a personal, opinionated essay rather than a systematic empirical study.

Significance. If the central causal claim—that publishing a baseline plus data/scripts is sufficient to create a productive research community—were established, it would be a practically important and broadly applicable recipe for accelerating research. The retrospective also documents a historically influential dataset collection (PROMISE) and a widely cited paper, and it makes a falsifiable prediction that the same mechanism can be reused for new topics. The paper's strengths include naming a concrete case, providing some external indicators of impact (citation counts, an alleged 20% adoption in leading TSE papers), and being transparent about the author's own role. However, the evidence presented is largely anecdotal and self-referential: the causal claim rests on a single case study whose confounds are not examined, the 20% statistic is not accompanied by a reproducible methodology, and the nine laws are generalizations from the author's own experience and self-cited studies. As a personal retrospective, the narrative is coherent and readable, but as a scientific argument for the universal mechanism, it is not yet supported.

major comments (4)
  1. [Section 1] The central claim, 'Industry can get any research it wants, just by publishing a baseline result along with the data and scripts needed to reproduce that work,' is causal and general, but it is supported only by a single case study with several confounds: the 2007 paper appeared in a top journal (TSE), was the first public defect-prediction baseline in that venue, benefited from the PROMISE conference and repository infrastructure (including student sprints and a steering committee), and was promoted by the author himself. The manuscript does not consider or control for these alternative explanations, nor does it offer any counterfactual or matched comparison. Please either temper the claim to a hypothesis or personal observation, or add a systematic comparison (e.g., similar papers that shared data but did not create communities, or communities formed without shared baselines).
  2. [Section 2] The assertion that 'By 2018, twenty percent of leading TSE papers (according to Google Scholar Metrics), incorporated artifacts introduced and disseminated by this research' is load-bearing for the impact narrative, but no methodology is provided. How were 'leading TSE papers' selected and counted? What counts as 'incorporated artifacts'—citing PROMISE data, using the 2007 data, or using any data from the PROMISE repository? Is the count reproducible? Without a precise definition and derivation, this statistic cannot be verified and should not be presented as an established fact.
  3. [Section 5.2] The nine 'Menzies's Laws' are presented as general empirical laws, but they are each supported by anecdotal evidence, often from the author's own papers. For example, 'Menzies's 6th Law: Data quality matters less than you think' is based on a single mutation study described in the text, and 'Menzies's 5th Law: Bigger is not necessarily better' is inferred from a systematic review that only 13/229 LLM papers compared to other methods—which does not itself demonstrate that smaller models are better. These statements are better framed as personal reflections or working hypotheses, with explicit caveats about the limited evidence, rather than as laws.
  4. [Section 5.1] The four-phase model of shared-data lifecycles ('Data? Good luck with that!' through 'A graveyard of progress') is presented as a general pattern, but it is based on the PROMISE experience alone. The claim that PROMISE data eventually became a 'lead weight' that stifled research is not supported by evidence; the examples of papers reusing decades-old datasets could be interpreted as evidence of continued utility, not stagnation. The author's own editorial decision to desk-reject papers using his 2005 datasets does not establish a field-wide problem. Please either present this as a personal narrative or provide systematic evidence of the claimed stagnation and its causes.
minor comments (4)
  1. [Throughout] The manuscript contains numerous typos and grammatical errors that should be corrected: 'scripts need to reproduce' (Abstract), 'Those result were' (Abstract), 'halycon' (Section 1), 'wore no suite and tie' (Section 1), 'gather all can that be collected' (Section 2), and 'more one attribute' (Section 2).
  2. [Section 1] The statement that the PROMISE repository 'grew so large and that it we moved it to the Large Hadron Collider' is confusing and appears to be a joke or error; if it is a reference to the 'Seacraft' data at Zenodo, clarify the wording.
  3. [References] Several references are self-citations (e.g., [24] duplicates [1]; [7], [23], [34], [39], [43], [44], [50], [57], [59], [61], [62], [64], [65], [66], [69], [70], [71], [72], [73], [77], [80], [85], [86], [87], [88]), which is understandable in a retrospective but should be presented carefully so that the independence of the evidence is clear.
  4. [Section 5.2] The 'Menzies's 3rd Law: Turkish toasters can predict for errors in deep space satellites' is stated without a reference to the transfer-learning study [67] in the surrounding text, making it hard to evaluate; add a citation and a brief explanation of the study design.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the retrospective's causal generalization is under-evidenced, but no prediction or derivation reduces to its own inputs by construction.

full rationale

This is a retrospective essay, not a formal derivation. The central claim—that publishing a baseline result plus data and scripts can seed a research community—is supported by the historical case of the author's 2007 TSE paper and the PROMISE repository. The evidence includes externally checkable facts: Google Scholar citation counts and the 20% adoption figure for TSE papers. Those facts are independent of the paper's own argument, not fitted parameters or conclusions assumed by definition. The author's role as creator of PROMISE and co-author of the 2007 paper makes the case self-referential as an example, but the paper does not define the outcome in terms of the treatment, does not import a self-authored uniqueness theorem, and does not smuggle an ansatz via citation. Confounds such as venue prestige, first-mover advantage, and the PROMISE conference/repository scaffolding are plausible threats to the causal inference's validity, but they are evidence-quality concerns, not circularity: they do not make the conclusion equivalent to its input by construction. No quoted equation, fitted parameter, or citation chain reduces the paper's central claim to its own premises, so no circular step meets the evidentiary standard.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper has no fitted parameters and no new postulated mechanisms such as new entities, forces, or particles. It does, however, rely on several unproven assumptions about the accuracy and causal interpretation of its historical statistics and on the representativeness of the author's personal experience.

assumptions (4)
  • domain assumption The 20% of leading TSE papers using PROMISE-era data, derived from Google Scholar Metrics, is accurate and methodologically comparable.
    Stated in Section 1 without a search protocol, journal list, or inclusion criteria; the statistic supports the paper's impact narrative.
  • domain assumption Citation counts and adoption in published papers are valid proxies for research impact and community formation.
    Used throughout Sections 1 and 3 to equate popularity and citations with influence and progress.
  • ad hoc to paper The availability of data and scripts caused the community's reproducibility practices, rather than merely coinciding with them.
    This is the core thesis of the paper; no counterfactual, control group, or competing explanation is offered.
  • ad hoc to paper The author's own research experience and self-cited studies are representative enough to support nine general laws about software engineering data.
    Section 5.2 generalizes from personal projects and self-cited papers without a systematic survey of the field.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Retrospective: Data Mining Static Code Attributes to Learn Defect Predictors." pith.science (2026). https://pith.science/paper/WAQQDLK5

@misc{pith2026250115662,
  author       = {Pith},
  title        = {Pith review of: Retrospective: Data Mining Static Code Attributes to Learn Defect Predictors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WAQQDLK5}},
  note         = {Machine review of arXiv:2501.15662}
}
read the original abstract

Industry can get any research it wants, just by publishing a baseline result along with the data and scripts need to reproduce that work. For instance, the paper ``Data Mining Static Code Attributes to Learn Defect Predictors'' presented such a baseline, using static code attributes from NASA projects. Those result were enthusiastically embraced by a software engineering research community, hungry for data. At its peak (2016) this paper was SE's most cited paper (per month). By 2018, twenty percent of leading TSE papers (according to Google Scholar Metrics), incorporated artifacts introduced and disseminated by this research. This brief note reflects on what we should remember, and what we should forget, from that paper.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

93 extracted references · 73 canonical work pages

  1. [1]

    Data mining static code attributes to learn defect predictors,

    T. Menzies, J. Greenwald, and A. Frank, “Data mining static code attributes to learn defect predictors,” IEEE Transactions on Software Engineering, vol. 33, no. 1, pp. 2–13, 2007

    2007

  2. [2]

    A brief note, with thanks, on the contributions of guenther ruhe,

    T. Menzies, “A brief note, with thanks, on the contributions of guenther ruhe,” Information and Software Technology , vol. 173, p. 107486, 2024. [Online]. Available: https://www.sciencedirect. com/science/article/pii/S0950584924000910

    2024

  3. [3]

    The road ahead for mining software repositories,

    A. E. Hassan, “The road ahead for mining software repositories,” 2008 Frontiers of Software Maintenance , pp. 48–57, 2008. [Online]. Available: https://api.semanticscholar.org/CorpusID:8656663

    2008

  4. [4]

    Foreword,

    P . Devanbu, “Foreword,” in Sharing Data and Models in Software Engineering, T. Menzies, E. Kocag ¨uneli, L. Minku, F. Peters, and B. Turhan, Eds. Morgan Kaufmann, 2015, pp. vii–viii

    2015

  5. [5]

    Replicating MSR: A study of the potential replicability of papers published in the mining software repositories proceed- ings,

    G. Robles, “Replicating MSR: A study of the potential replicability of papers published in the mining software repositories proceed- ings,” in 2010 7th IEEE Working Conference on Mining Software Repositories (MSR 2010). IEEE, 2010, pp. 171–180

    2010

  6. [6]

    Revisiting the repro- ducibility of empirical software engineering studies based on data retrieved from development repositories,

    J. M. Gonzalez-Barahona and G. Robles, “Revisiting the repro- ducibility of empirical software engineering studies based on data retrieved from development repositories,” Information and Software Technology, vol. 164, p. 107318, 2023

    2023

  7. [7]

    Tuning for software analytics: Is it really necessary?

    W. Fu, T. Menzies, and X. Shen, “Tuning for software analytics: Is it really necessary?” IST, 2016

    2016

  8. [8]

    Common trends in software fault and failure data,

    M. Hamill and K. Goseva-Popstojanova, “Common trends in software fault and failure data,” TSE, 2009

    2009

Show all 93 references
  1. [9]

    An investigation into the functional form of the size-defect relationship for software modules,

    A. G. Koru, D. Zhang, K. El Emam, and H. Liu, “An investigation into the functional form of the size-defect relationship for software modules,” TSE, 2009

    2009

  2. [10]

    Where the bugs are,

    T. J. Ostrand, E. J. Weyuker, and R. M. Bell, “Where the bugs are,” in ACM SIGSOFT Software Engineering Notes, 2004

    2004

  3. [11]

    Ai-based software defect predictors: Applications and benefits in a case study,

    A. T. Misirli, A. Bener, and R. Kale, “Ai-based software defect predictors: Applications and benefits in a case study,”AI Magazine, 2011

    2011

  4. [12]

    Software measurement: a necessary scientific basis,

    N. Fenton, “Software measurement: a necessary scientific basis,” IEEE Transactions on Software Engineering , vol. 20, no. 3, pp. 199– 206, 1994

    1994

  5. [13]

    A complexity measure,

    T. J. McCabe, “A complexity measure,” TSE, no. 4, pp. 308–320, 1976

    1976

  6. [14]

    M. H. Halstead, Elements of software science . Elsevier New York, 1977, vol. 7

    1977

  7. [15]

    N. E. Fenton and S. Pfleeger, Software Metrics: A Rigorous and Practical Approach. International Thompson Press, 1997

    1997

  8. [16]

    A critique of three metrics,

    M. Shepperd and D. Ince, “A critique of three metrics,” Journal of Systems and Software, vol. 26, no. 3, pp. 197–210, September 1994

    1994

  9. [17]

    When less is more: on the value of “co-training

    S. Majumder, J. Chakraborty, and T. Menzies, “When less is more: on the value of “co-training” for semi-supervised software defect predictors,” Empirical Software Engineering, vol. 29, no. 2, Feb. 2024

    2024

  10. [18]

    Comparing static bug finders and statistical prediction,

    F. Rahman, S. Khatri, E. T. Barr, and P . Devanbu, “Comparing static bug finders and statistical prediction,” in ICSE. ACM, 2014

    2014

  11. [19]

    Percep- tions, expectations, & challenges in defect prediction,

    Z. Wan, X. Xia, A. E. Hassan, D. Lo, J. Yin, and X. Yang, “Percep- tions, expectations, & challenges in defect prediction,” TSE, 2018

    2018

  12. [20]

    Remi: defect prediction for efficient api testing,

    M. Kim, J. Nam, J. Yeon, S. Choi, and S. Kim, “Remi: defect prediction for efficient api testing,” in FSE. ACM, 2015

    2015

  13. [21]

    Can traditional fault prediction models be used for vulnerability prediction?

    Y. Shin and L. Williams, “Can traditional fault prediction models be used for vulnerability prediction?” EMSE, 2013. [Online]. Available: https://doi.org/10.1007/s10664-011-9190-8

    2013 doi

  14. [22]

    Putting it all together: Using socio-technical networks to predict failures,

    C. Bird, N. Nagappan, H. Gall, B. Murphy, and P . Devanbu, “Putting it all together: Using socio-technical networks to predict failures,” in ISSRE, 2009

    2009

  15. [23]

    Defect prediction from static code features: Current results, limi- tations, new approaches,

    T. Menzies, Z. Milton, B. Turhan, B. Cukic, Y. Jiang, and A. Bener, “Defect prediction from static code features: Current results, limi- tations, new approaches,” ASE, 2010

    2010

  16. [24]

    Data mining static code attributes to learn defect predictors,

    T. Menzies, J. Greenwald, and A. Frank, “Data mining static code attributes to learn defect predictors,” TSE, 2007

    2007

  17. [25]

    An extensive compari- son of bug prediction approaches,

    M. D’Ambros, M. Lanza, and R. Robbes, “An extensive compari- son of bug prediction approaches,” in MSR, 2010

    2010

  18. [26]

    Use of relative code churn measures to predict system defect density,

    N. Nagappan and T. Ball, “Use of relative code churn measures to predict system defect density,” in ICSE, 2005

    2005

  19. [27]

    Code churn: A measure for estimating the impact of code change,

    S. Elbaum and J. Munson, “Code churn: A measure for estimating the impact of code change,” ICSME, 2000

    2000

  20. [28]

    A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction,

    R. Moser, W. Pedrycz, and G. Succi, “A comparative analysis of the efficiency of change metrics and static code attributes for defect prediction,” in 30th ICSE, 2008

    2008

  21. [29]

    Predicting faults using the complexity of code changes,

    A. E. Hassan, “Predicting faults using the complexity of code changes,” in IEEE 31st ICSE, 2009

    2009

  22. [30]

    Defect prediction: Accomplishments and future challenges,

    Y. Kamei and E. Shihab, “Defect prediction: Accomplishments and future challenges,” in SANER, 2016

    2016

  23. [31]

    A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each,

    C. Le Goues, M. Dewey-Vogt, S. Forrest, and W. Weimer, “A systematic study of automated program repair: Fixing 55 out of 105 bugs for $8 each,” ser. ICSE, 2012

    2012

  24. [32]

    A practical guide for using statistical tests to assess randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in ICSE, 2011

    2011

  25. [33]

    Applications of psychological science for actionable analytics,

    D. Chen, W. Fu, R. Krishna, and T. Menzies, “Applications of psychological science for actionable analytics,” in FSE, 2018

    2018

  26. [34]

    Is better data better than better data miners?: on the benefits of tuning smote for defect prediction,

    A. Agrawal and T. Menzies, “Is better data better than better data miners?: on the benefits of tuning smote for defect prediction,” in IST. ACM, 2018

    2018

  27. [35]

    Automating change-level self-admitted technical debt determination,

    M. Yan, X. Xia, E. Shihab, D. Lo, J. Yin, and X. Yang, “Automating change-level self-admitted technical debt determination,” TSE, 2018

    2018

  28. [36]

    A large-scale empirical study of just-in-time quality assurance,

    Y. Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,” TSE, 2013

    2013

  29. [37]

    Clever: Combining code metrics with clone detection for just-in-time fault prevention and resolution in large industrial projects,

    M. Nayrolles and A. Hamou-Lhadj, “Clever: Combining code metrics with clone detection for just-in-time fault prevention and resolution in large industrial projects,” in MSR, 2018

    2018

  30. [38]

    Commit guru: Analytics and risk prediction of software commits,

    C. Rosen, B. Grawi, and E. Shihab, “Commit guru: Analytics and risk prediction of software commits,” ser. ESEC/FSE 2015, 2015

    2015

  31. [39]

    Bellwethers: A baseline method for transfer learning,

    R. Krishna and T. Menzies, “Bellwethers: A baseline method for transfer learning,” TSE, 2018

    2018

  32. [40]

    Heterogeneous defect prediction,

    J. Nam, W. Fu, S. Kim, T. Menzies, and L. Tan, “Heterogeneous defect prediction,” TSE, 2018

    2018

  33. [41]

    Revisiting the impact of classification techniques on the performance of defect prediction models,

    B. Ghotra, S. McIntosh, and A. E. Hassan, “Revisiting the impact of classification techniques on the performance of defect prediction models,” in 2015 37th ICSE

    2015

  34. [42]

    What is wrong with topic modeling? and how to fix it using search-based software engi- neering,

    A. Agrawal, W. Fu, and T. Menzies, “What is wrong with topic modeling? and how to fix it using search-based software engi- neering,” IST, 2018

    2018

  35. [43]

    Easy over hard: A case study on deep learning,

    W. Fu and T. Menzies, “Easy over hard: A case study on deep learning,” in FSE, 2017

    2017

  36. [44]

    Why is differential evolution better than grid search for tuning defect predictors?

    W. Fu, V . Nair, and T. Menzies, “Why is differential evolution better than grid search for tuning defect predictors?” CoRR, 2016

    2016

  37. [46]

    Software engineering economics,

    B. W. Boehm, “Software engineering economics,” 1981

    1981

  38. [47]

    Statistical analysis on the productivity of data processing with development projects using the function point technique,

    J.-M. Desharnais, “Statistical analysis on the productivity of data processing with development projects using the function point technique,” Universit´ e du Qu´ ebec ` a Montr´ eal, 1988

    1988

  39. [48]

    How good is your blind spot sampling policy,

    T. Menzies and J. S. Di Stefano, “How good is your blind spot sampling policy,” in HASE’04. IEEE, 2004, pp. 129–138

    2004

  40. [49]

    Towards identifying software project clusters with regard to defect prediction,

    M. Jureczko and L. Madeyski, “Towards identifying software project clusters with regard to defect prediction,” in PROMISE’10, ser. PROMISE ’10, 2010

    2010

  41. [50]

    Sequential model op- timization for software effort estimation,

    T. Xia, R. Shu, X. Shen, and T. Menzies, “Sequential model op- timization for software effort estimation,” IEEE Transactions on Software Engineering, vol. 48, no. 6, pp. 1994–2009, 2022

    1994

  42. [51]

    Commit guru: analytics and risk prediction of software commits,

    C. Rosen, B. Grawi, and E. Shihab, “Commit guru: analytics and risk prediction of software commits,” in FSE’15, 2015, pp. 966–969

    2015

  43. [52]

    Repro- ducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the szz algorithm,

    G. Rodriguez, G. Robles, and J. Gonzalez-Barahona, “Repro- ducibility and credibility in empirical software engineering: A case study based on a systematic literature review of the use of the szz algorithm,” IST, 2018

    2018

  44. [53]

    Problems with szz and features: An empirical study of the state of practice of defect prediction data collection,

    S. Herbold, A. Trautsch, F. Trautsch, and B. Ledel, “Problems with szz and features: An empirical study of the state of practice of defect prediction data collection,” Empirical Software Engineering , vol. 27, no. 2, p. 42, 2022

    2022

  45. [54]

    Deeplinedp: Towards a deep learning approach for line-level defect prediction,

    C. Pornprasit and C. K. Tantithamthavorn, “Deeplinedp: Towards a deep learning approach for line-level defect prediction,” IEEE Transactions on Software Engineering, vol. 49, no. 1, pp. 84–98, 2023. 6

    2023

  46. [55]

    Explainable ai for software engineering,

    C. K. Tantithamthavorn and J. Jiarpakdee, “Explainable ai for software engineering,” in ASE’21. IEEE, 2021, pp. 1–2

    2021

  47. [56]

    Fairway: a way to build fair ml software,

    J. Chakraborty, S. Majumder, Z. Yu, and T. Menzies, “Fairway: a way to build fair ml software,” in FSE’28, 2020, pp. 654–665

    2020

  48. [57]

    Don’t lie to me: Avoiding malicious explanations with stealth,

    L. Alvarez and T. Menzies, “Don’t lie to me: Avoiding malicious explanations with stealth,” IEEE Software, vol. 40, no. 3, pp. 43–53, 2023

    2023

  49. [58]

    Converging on the Optimal Attain- ment of Requirements,

    M. S. Feather and T. Menzies, “Converging on the Optimal Attain- ment of Requirements,” in RE’02. IEEE Computer Society, 2002, pp. 263–272

    2002

  50. [59]

    How to avoid drastic software process change (using stochastic stability),

    T. Menzies, S. Williams, O. El-Rawas, B. Boehm, and J. Hihn, “How to avoid drastic software process change (using stochastic stability),” in 2009 IEEE 31st International Conference on Software Engineering, 2009, pp. 540–550

    2009

  51. [60]

    The business case for automated software engineer- ing,

    T. Menzies, O. Elrawas, J. Hihn, M. Feather, R. Madachy, and B. Boehm, “The business case for automated software engineer- ing,” in ASE’07, 2007, p. 303–312

    2007

  52. [61]

    Replication can improve prior results: A github study of pull request acceptance,

    D. Chen, K. T. Stolee, and T. Menzies, “Replication can improve prior results: A github study of pull request acceptance,” in ICPC, 2019

    2019

  53. [62]

    Learning from very little data: On the value of landscape analysis for predicting software project health,

    A. Lustosa and T. Menzies, “Learning from very little data: On the value of landscape analysis for predicting software project health,” TOSEM, vol. 33, no. 3, pp. 1–22, 2024

    2024

  54. [63]

    Finding faster configurations using flash,

    V . Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,” IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 794–811, 2020

    2020

  55. [64]

    Gale: Geometric active learning for search-based software engineering,

    J. Krall, T. Menzies, and M. Davies, “Gale: Geometric active learning for search-based software engineering,” IEEE Transactions on Software Engineering, vol. 41, no. 10, pp. 1001–1018, 2015

    2015

  56. [65]

    Finding better active learn- ers for faster literature reviews,

    Z. Yu, N. A. Kraft, and T. Menzies, “Finding better active learn- ers for faster literature reviews,” Empirical Software Engineering , vol. 23, pp. 3161–3186, 2018

    2018

  57. [66]

    Frugal: unlocking semi-supervised learn- ing for software analytics,

    H. Tu and T. Menzies, “Frugal: unlocking semi-supervised learn- ing for software analytics,” in ASE’22. IEEE Press, 2022, p. 394–406

    2022

  58. [67]

    On the relative value of cross-company and within-company data for defect prediction,

    B. Turhan, T. Menzies, A. B. Bener, and J. Di Stefano, “On the relative value of cross-company and within-company data for defect prediction,” Empirical Software Engineering, vol. 14, pp. 540– 578, 2009

    2009

  59. [68]

    A survey on transfer learning,

    S. J. Pan and Q. Yang, “A survey on transfer learning,” IEEE Transactions on knowledge and data engineering , vol. 22, no. 10, pp. 1345–1359, 2009

    2009

  60. [69]

    Shockingly simple:

    T. Menzies, “Shockingly simple:” keys” for better ai for se,” IEEE Software, vol. 38, no. 2, pp. 114–118, 2021

    2021

  61. [70]

    Better predictors for issue lifetime,

    M. Rees-Jones, M. Martin, and T. Menzies, “Better predictors for issue lifetime,” arXiv preprint arXiv:1702.07735, 2017

    2017 arXiv

  62. [71]

    Ac- tive learning and effort estimation: Finding the essential content of software effort estimation data,

    E. Kocaguneli, T. Menzies, J. Keung, D. Cok, and R. Madachy, “Ac- tive learning and effort estimation: Finding the essential content of software effort estimation data,” TSE, vol. 39, no. 8, pp. 1040–1053, 2013

    2013

  63. [72]

    Lace2: Better privacy- preserving data sharing for cross project defect prediction,

    F. Peters, T. Menzies, and L. Layman, “Lace2: Better privacy- preserving data sharing for cross project defect prediction,” in 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, vol. 1, 2015, pp. 801–811

    2015

  64. [73]

    Finding the right data for software cost modeling,

    Z. Chen, T. Menzies, D. Port, and D. Boehm, “Finding the right data for software cost modeling,” IEEE software, vol. 22, no. 6, pp. 38–46, 2005

    2005

  65. [74]

    Semi-supervised learning literature survey,

    X. Zhu, “Semi-supervised learning literature survey,” Computer Sciences Technical Report, vol. 1530, pp. 1–59, 2005

    2005

  66. [75]

    Extensions of lipschitz map- pings into a hilbert space,

    W. B. Johnson and J. Lindenstrauss, “Extensions of lipschitz map- pings into a hilbert space,” Contemporary Mathematics, vol. 26, pp. 189–206, 1984

    1984

  67. [76]

    A comprehensive comparative study of clustering- based unsupervised defect prediction models,

    Z. Xu, L. Li, M. Yan, J. Liu, X. Luo, J. Grundy, Y. Zhang, and X. Zhang, “A comprehensive comparative study of clustering- based unsupervised defect prediction models,” Journal of Systems and Software, vol. 172, p. 110862, 2021

    2021

  68. [77]

    Implications of ceiling effects in defect predictors,

    T. Menzies, B. Turhan, A. Bener, G. Gay, B. Cukic, and Y. Jiang, “Implications of ceiling effects in defect predictors,” in Proceedings of the 4th international workshop on Predictor models in software engineering, 2008, pp. 47–54

    2008

  69. [78]

    Why power laws? an explanation from fine-grained code changes,

    Z. Lin and J. Whitehead, “Why power laws? an explanation from fine-grained code changes,” in 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories. IEEE, 2015, pp. 68–75

    2015

  70. [79]

    On the naturalness of software,

    A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P . Devanbu, “On the naturalness of software,” in Proceedings of the 34th International Conference on Software Engineering, ser. ICSE ’12. IEEE Press, 2012, p. 837–847

    2012

  71. [80]

    ”sampling

    J. Chen, V . Nair, R. Krishna, and T. Menzies, “”sampling” as a baseline optimizer for search-based software engineering,” IEEE Transactions on Software Engineering (pre-print), pp. 1–1, 2018

    2018

  72. [81]

    Large language models for software engineering: A systematic literature review,

    X. Hou, Y. Zhao, Y. Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engineering: A systematic literature review,” ACM Trans. Softw. Eng. Methodol. , vol. 33, no. 8, Dec. 2024. [Online]. Available: https://doi.org/10.1145/3695988

    2024 doi

  73. [82]

    Why do tree-based models still outperform deep learning on typical tabular data?

    L. Grinsztajn, E. Oyallon, and G. Varoquaux, “Why do tree-based models still outperform deep learning on typical tabular data?” in NeurIPS’22, 2022

    2022

  74. [83]

    A survey on deep tabular learning,

    S. Somvanshi, S. Das, S. A. Javed, G. Antariksa, and A. Hossain, “A survey on deep tabular learning,” arXiv preprint arXiv:2410.12034, 2024

    2024 arXiv

  75. [84]

    Agile effort estimation: Have we solved the problem yet? insights from a replication study,

    V . Tawosi, R. Moussa, and F. Sarro, “Agile effort estimation: Have we solved the problem yet? insights from a replication study,” IEEE Transactions on Software Engineering , vol. 49, no. 4, pp. 2677– 2697, 2023

    2023

  76. [85]

    500+ times faster than deep learning,

    S. Majumder, N. Balaji, K. Brey, W. Fu, and T. Menzies, “500+ times faster than deep learning,” in Proceedings of the 15th International Conference on Mining Software Repositories. ACM, 2018

    2018

  77. [86]

    Trading off scalability, privacy, and performance in data synthesis,

    X. Ling, T. Menzies, C. Hazard, J. Shu, and J. Beel, “Trading off scalability, privacy, and performance in data synthesis,” IEEE Access, vol. 12, pp. 26 642–26 654, 2024

    2024

  78. [87]

    Easy over hard: a case study on deep learning,

    W. Fu and T. Menzies, “Easy over hard: a case study on deep learning,” in Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering , ser. ESEC/FSE 2017. New York, NY, USA: Association for Computing Machinery, 2017, p. 49–60. [Online]. Available: https:...

    2017 doi

  79. [88]

    Ai over-hype: A dangerous threat (and how to fix it),

    B. Johnson and T. Menzies, “Ai over-hype: A dangerous threat (and how to fix it),” IEEE Software , vol. 41, no. 6, pp. 131–138, 2024

    2024

  80. [89]

    Data quality: Some comments on the nasa software defect datasets,

    M. Shepperd, Q. Song, Z. Sun, and C. Mair, “Data quality: Some comments on the nasa software defect datasets,” IEEE Transactions on software engineering, vol. 39, no. 9, pp. 1208–1215, 2013

    2013

  81. [90]

    Using bad learners to find good configurations,

    V . Nair, T. Menzies, N. Siegmund, and S. Apel, “Using bad learners to find good configurations,” in 11th Joint Meeting on FSE. ACM, 2017

    2017

  82. [91]

    How to ”dodge

    A. Agrawal, W. Fu, D. Chen, X. Shen, and T. Menzies, “How to ”dodge” complex software analytics?” TSE, 2019

    2019

  83. [92]

    Automated parameter optimization of classification techniques for defect prediction models,

    C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Mat- sumoto, “Automated parameter optimization of classification techniques for defect prediction models,” in 38th ICSE, 2016

    2016

  84. [93]

    Tuning for software analytics,

    W. Fu, T. Menzies, and X. Shen, “Tuning for software analytics,” Inf. Softw. Technol., vol. 76, no. C, p. 135–146, Aug. 2016. [Online]. Available: https://doi.org/10.1016/j.infsof.2016.04.017

    2016 doi

  85. [94]

    P . R. Cohen, Empirical methods for artificial intelligence . MIT press Cambridge, MA, 1995. Tim Menzies (ACM Fellow, IEEE Fellow, ASE Fellow, Ph.D., UNSW, 1995) is a full Professor in Computer Science at North Carolina State. He is the director of the Irrational Research lab ...

    1995

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.