Pith. sign in

REVIEW 2 major objections 5 minor 63 references

Is Model Instability just Noise to be Tolerated or a Property that can be Managed?

T0 review · 2 major / 5 minor · reviewed 2026-07-14 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Model instability in software optimization is not noise to ignore but a property that can be measured and managed.

desk verdict Large-scale, careful empirical study that turns SBSE instability into a measurable evaluation axis and shows simple learner knobs improve both stability and quality. read the letter →

arxiv 2607.10420 v1 pith:JTPPH6IY submitted 2026-07-11 cs.SE cs.LG

Amirali Rayegan , Lunxiao Li , Tim Menzies This is my paper
classification cs.SEcs.LG
keywords modelinstabilitysearch-basedsoftwareengineeringanalyticsRashomoneffectmulti-objectiveoptimizationactivelearningperformancestabilityreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Rerunning the same multi-objective software optimizer often yields different recommendations, which erodes trust. Across 127 SE optimization problems and 12,700 test cases, a state-of-the-art active learner under default settings agrees on only about 3% of held-out cases. By changing how labels are spent, how candidates are chosen, how deep trees grow, and how splits are scored, agreement rises 4.8 times, the spread of optimization error falls by 22%, and recommendation quality improves rather than declines. Structural diversity of the trees stays high, so the paper targets consistency of recommendations, not identical model forms. Causal filtering and locality-based clustering help only partially, which the authors read as a residual floor set by the data itself—noise, scarce labels, proxy objectives, and many near-equivalent models. They conclude that instability should be reported as a standard evaluation axis alongside performance.

What carries the argument

Performance stability, measured by cross-run agreement of predicted scores on fixed hold-out rows and by the standard deviation of optimization error; the refined configuration of labeling budget, near-centroid acquisition, minimum leaf size, and Gini splitting is the lever that improves both stability and quality at once.

What would settle it

Run the same 20-seed agreement protocol with a different optimizer family on a large non-MOOT suite; if stability levers like those recommended here produce no agreement gain, or if another method drives near-full agreement with no quality loss, the claim of a data-inherent floor fails.

Watch

Extended reading notes

Core claim

Across 127 multi-objective SE optimization tasks, refined learner settings raise test-case agreement 4.8 times (364 to 1,740 at the main threshold), cut mean optimization-error standard deviation by 22% (17.4 to 13.6), and are statistically top-ranked for quality on 119 of 127 datasets versus 74 for defaults. Structural and performance instability are distinct phenomena; further causal and data-locality interventions leave a residual stability floor that the authors attribute to properties of the data rather than the learner alone.

Load-bearing premise

The residual stability floor and the recommended settings are assumed to generalize from one active-learning tree family and one suite of 127 multi-objective SE tasks to broader search-based software engineering.

Editorial extensions

If this is right

  • SE optimization tools should drop unstable defaults such as entropy splitting and pure exploit acquisition in favor of more stable choices like Gini and near acquisition.
  • Empirical SE papers should report agreement rates or error spread alongside mean optimization quality so practitioners can calibrate trust in any single run.
  • Forcing identical model structure is the wrong goal; recommendation consistency is the actionable target given many near-equivalent models.
  • The refined configuration and measured residual floor become a quantitative baseline for future work that tries to reduce SBSE instability.
  • Stability should be treated as a first-class evaluation axis, not an afterthought once mean performance looks good.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same labeling, acquisition, complexity, and split levers may stabilize other tree-based or active-learning optimizers outside software engineering.
  • If the residual floor is truly data-driven, cleaner labels and less noisy proxy objectives may raise the ceiling more than further learner tweaks.
  • Agreement-rate reporting could become a routine reproducibility metric for any stochastic software-analytics pipeline, not only multi-objective optimizers.
  • Domains rich in organizational confounders may still gain more from causal screening than the average task in this suite did.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper argues that performance instability in multi-objective SE optimization is a measurable, partially manageable property rather than mere noise. Across 127 MOOT tasks (12 700 test cases) and 20 random seeds, default EZR settings produce agreement on only 2.9 % of cases at α = 0.35. A refined configuration (label budget 50, near acquisition, min-leaf 3, Gini splits) raises agreement 4.8 imes to 13.7 %, cuts mean optimization-error standard deviation by 22 % (17.4 → 13.6), and is statistically top-ranked on 119/127 datasets versus 74 for defaults, without quality loss. Structural (weighted Jaccard) instability remains high, consistent with the Rashomon effect. Causal confounder filtering and locality-based clusterers improve stability only partially, leaving a residual floor attributed to data properties (noise, scarce labels, proxy objectives, large near-optimal model sets). The authors conclude that instability should be reported as a standard evaluation axis alongside performance.

Significance. If the empirical results hold, the work supplies a concrete, reproducible baseline for treating instability as a first-class concern in SBSE. Strengths include the scale (127 real multi-objective SE tasks), the dual measurement of structural versus performance stability, the factor-by-factor then joint validation of the refined settings (Table IV, Figs. 1, 6, 8, 9), the α-sweep, the KS + Cliff’s-δ ranking procedure, and the open reproduction package. Distinguishing performance agreement (actionable for practitioners) from structural diversity (largely irreducible) is a useful conceptual contribution. The residual-floor claim, while scoped as evidence rather than proof, usefully reframes future work toward data-side limits rather than purely algorithmic fixes.

major comments (2)
  1. [Abstract, RQ3, §V, Conclusion] The residual data-inherent floor (RQ3) and the claim that the refined settings form a baseline for SBSE more broadly rest on a single primary learner family (EZR) plus three clusterers evaluated on the MOOT collection. Section V correctly flags external validity, yet the abstract and conclusion phrase the contribution as applying to “SE optimization” / “SBSE instability” without equally prominent scoping. A short additional experiment with one non-tree optimizer, or a more explicit limitation statement in the abstract, would make the generalization claim load-bearing-safe.
  2. [§III-E, Eq. (7), Fig. 6, RQ0] Agreement (Eq. 7) is defined with a free threshold α whose primary reported value is 0.35. Figure 6 shows the ranking is robust across the sweep, which is good, but the headline 4.8 imes factor and the 13.7 % figure are α-specific. The paper should state more clearly why 0.35 is the preferred operating point (e.g., correspondence to a conventional effect-size band) rather than leaving it as an arbitrary but swept constant.
minor comments (5)
  1. [Fig. 2, Fig. 5] Figure 2 and Figure 5 trees are hard to read at column width; consider a compact side-by-side layout or an appendix with larger renderings.
  2. [Table III] Table III is dense; a short summary row or a supplementary breakdown by category size would help readers gauge diversity without scrolling.
  3. [§III-B, Fig. 9] The win metric (Eqs. 3–4) normalizes by median–min of d2h; a one-sentence reminder that this can compress differences on already-easy datasets would aid interpretation of the left-hand side of Figure 9.
  4. [RQ0–RQ1] A few forward references (e.g., “refined settings later recommended by RQ2”) appear before the settings are defined; a brief forward pointer or early table would reduce reader friction.
  5. [Table II, Table IV] Minor typographic inconsistencies: “min leaf size = [1, 3, 5, 7, 9]” versus later “min leaf=3”; unify notation.

Circularity Check

0 steps flagged · score 1.0 of 10

Empirical measurement study; no prediction or first-principles claim reduces to its inputs by construction. Minor self-citations supply the learner and data suite but are not load-bearing for the instability results.

full rationale

The paper’s central claims are empirical counts and statistical rankings: agreement rates (Eq. 7, α-sweep in Fig. 6), optimization-error standard deviations (Fig. 1), and top-rank win counts under KS+Cliff’s-δ (Figs. 8–9, Table IV). These are obtained by re-running EZR (and three clusterers) 20 times on 127 hold-out splits drawn from the MOOT collection and comparing predictions to true hold-out optima (Eqs. 3–5). Nothing is defined in terms of the quantity being measured, no parameter is fitted to a subset and then “predicted” on a near-identical quantity, and no uniqueness theorem or ansatz is imported to force the result. The refined configuration (50 labels, near, min-leaf=3, Gini) is selected by one-factor-at-a-time experiments on the same suite and then jointly re-evaluated; that is ordinary experimental practice, not circular reduction. Self-citations to prior EZR and MOOT papers ([34], [52], [54], [43]) merely identify the learner and data source; the instability numbers, the 4.8 imes agreement lift, the 22 % std reduction, and the residual-floor observation are newly computed here and are externally falsifiable via the released reproduction package. Structural-versus-performance decoupling (RQ1) is likewise an observed contrast between weighted Jaccard (Eq. 6) and prediction agreement, not a definitional identity. Consequently the derivation chain contains no circular step of the kinds enumerated; the modest score of 1 reflects only the presence of non-load-bearing self-citations that supply experimental materials.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on a small set of free measurement choices (α, budgets, leaf sizes) and standard domain assumptions about multi-objective SE data and the Rashomon effect; no new physical or mathematical entities are invented. The residual-floor argument inherits the Rashomon-ratio literature rather than deriving it.

free parameters (4)
  • agreement threshold α = 0.35 (primary)
    Models ‘agree’ when σ_models < α × σ_data; primary results reported at α=0.35 (swept 0.15–1.0).
  • labeling budget n1 = 50 (refined)
    Number of labels spent before tree construction; swept [10,20,50,100,200], refined setting = 50.
  • min leaf size = 3 (refined)
    Controls tree complexity; swept [1,3,5,7,9], refined = 3.
  • n2 recommendation labels = 10
    Top-n2 hold-out candidates labeled to pick final recommendation; fixed at default 10.
assumptions (4)
  • domain assumption Many near-optimal models exist for the same tabular SE data (Rashomon effect / large Rashomon ratio under label noise).
    Invoked in §II-B and RQ1 to explain why structural instability is largely irreducible; drawn from Breiman, Semenova, Xin, Rudin.
  • domain assumption Distance-to-heaven (d2h) and the derived win score are adequate proxies for multi-objective configuration quality.
    All performance and stability measures are computed from Model(row)→predicted win (§III-B).
  • standard math KS test (α=0.05) plus Cliff’s δ ≤ 0.195 plus top-set ranking correctly identify statistically superior treatments.
    §III-F statistical procedure used for all win counts.
  • domain assumption EZR’s active-learning tree pipeline is a representative state-of-the-art SBSE optimizer for the purposes of studying instability.
    §III-C justification; external-validity threat acknowledges limited learner family.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Is Model Instability just Noise to be Tolerated or a Property that can be Managed?." pith.science (2026). https://pith.science/paper/JTPPH6IY

@misc{pith2026260710420,
  author       = {Pith},
  title        = {Pith review of: Is Model Instability just Noise to be Tolerated or a Property that can be Managed?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JTPPH6IY}},
  note         = {Machine review of arXiv:2607.10420}
}
read the original abstract

In software analytics, rerunning the same analysis twice often yields different models and conclusions. This reduces trust in the model and limits its use. We find that model instability is a major problem. Across 127 multi-objective SE optimization problems (12,700 test cases), repeated runs of a state-of-the-art optimizer agree on only 13.7% of test cases, even under improved settings. We argue that this instability is not merely noise to tolerate, but a property that can be measured and managed. By adjusting how labels are spent, how complex the models become, and how splits are scored, we obtain models that agree 4.8 times as often as the default configuration. The standard deviation of optimization error falls by 22% on average (mean std 17.4 to 13.6), while recommendation quality improves rather than degrades. In terms of quality, the refined settings are statistically top-ranked on 119 of 127 datasets, compared to 74 for the defaults. We then test causal and data-locality interventions and find that they help only partially, suggesting a residual stability floor. Our evidence suggests there are fundamental limits to stability set by the data itself (noise, scarce labels, proxy objectives, and the many near-equivalent models a dataset admits). We conclude that instability should be treated as a standard evaluation axis in SE optimization, which should be routinely measured, reported alongside performance, and used to calibrate trust in any single run. The methods in this paper provide a baseline against which future efforts to reduce SBSE instability can be judged. To support open science, we offer the following reproduction package: https://tinyurl.com/Model-Instability

Figures

Figures reproduced from arXiv: 2607.10420 by the authors.

Figure 1
Figure 1. Performance instability across 127 SE datasets. Standard deviation of prediction errors as defined by Equation 4 (difference between the referenced optimal and the outcome of the model). Blue and red lines show, respectively, results obtained using (1) the initial default settings or (2) the settings recommended by this paper. Note that, with our methods (the red line), performance instability is nearly always reduc… view at source ↗
Figure 2
Figure 2. Structural instability. Example of models learned in this paper. Results are from 4 repeats varying only the random seed, extracted from [12], on Coc1000 dataset from MOOT (see Table III). as independent drivers, with a validated configuration that improves both stability and quality; and (4) evidence of a data￾inherent floor that learner-side and causal interventions reduce but cannot fully eliminate. To support op… view at source ↗
Figure 3
Figure 3. Coefficient instability in effort estimation [7]. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Overview of the EZR active-learning pipeline, from warm-start initialization through iterative acquisition and labeling [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Four trees trained on coc1000.csv (same data as Figure 2) with RQ2-refined settings and different random seeds [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Agreement counts (out of 12,700 test cases) across thresholds α ∈ [0.15, 1.0] for initial and refined EZR setting. Factor Default Recommended Labeling budget 20 50 Acquisition strategy Xploit near Min leaf size 2 3 Splitting criterion Entropy Gini TABLE IV: Suggested E…
Figure 7
Figure 7. Figure 7: Structural similarity across 20 trees per dataset under initial and refined EZR settings. Refined settings greatly improve performance stability ( [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Summary of four instability-cause experiments. Values indicate the percentage of datasets where each treatment achieved [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Performance results (win of optimization quality Q, defined in Equation 3) across 127 datasets, measured as average deviation from the referenced optimum (lower is better). Datasets sorted by initial performance. Refined settings are statistically top-ranked on 119 dat…
Figure 10
Figure 10. Figure 10: Performance (dataset statistical wins), stability (dataset statistical wins), and agreement (test-case count out of 12,700) for EZR and causal pipeline across all datasets [PITH_FULL_IMAGE:figures/full_fig_p010_10.png]
Figure 11
Figure 11. Figure 11: Comparing Performance and stability of EZR with [PITH_FULL_IMAGE:figures/full_fig_p010_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

63 extracted references · 3 linked inside Pith

  1. [1]

    Software analytics to software practice: a systematic literature review,

    T. M. Abdellatif, L. F. Capretz, and D. Ho, “Software analytics to software practice: a systematic literature review,” in2015 IEEE/ACM 1st International Workshop on Big Data Software Engineering. IEEE, 2015, pp. 30–36

    2015

  2. [2]

    Analyze this! 145 questions for data sci- entists in software engineering,

    A. Begel and T. Zimmermann, “Analyze this! 145 questions for data sci- entists in software engineering,” inProceedings of the 36th International Conference on Software Engineering, 2014, pp. 12–23

    2014

  3. [3]

    Software development analytics in practice: A systematic literature review: J. caldeira et al

    J. Caldeira, F. Brito e Abreu, J. Cardoso, R. Sim ˜oes, T. Oliveira, and J. Pereira dos Reis, “Software development analytics in practice: A systematic literature review: J. caldeira et al.”Archives of Computational Methods in Engineering, vol. 30, no. 3, pp. 2041–2080, 2023

  4. [4]

    Impact of software engineer- ing research in practice: A patent and author survey analysis,

    Z. Kotti, G. Gousios, and D. Spinellis, “Impact of software engineer- ing research in practice: A patent and author survey analysis,”IEEE Transactions on Software Engineering, vol. 49, no. 4, pp. 2020–2038, 2022

    2020

  5. [5]

    Software analytics in practice,

    D. Zhang, S. Han, Y . Dang, J.-G. Lou, H. Zhang, and T. Xie, “Software analytics in practice,”IEEE software, vol. 30, no. 5, pp. 30–37, 2013

    2013

  6. [6]

    An empirical study of log analysis at microsoft,

    S. He, X. Zhang, P. He, Y . Xu, L. Li, Y . Kang, M. Ma, Y . Wei, Y . Dang, S. Rajmohanet al., “An empirical study of log analysis at microsoft,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 1465–1476

    2022

  7. [7]

    Local versus global lessons for defect prediction and effort estimation,

    T. Menzies, A. Butcher, D. Cok, A. Marcus, L. Layman, F. Shull, B. Turhan, and T. Zimmermann, “Local versus global lessons for defect prediction and effort estimation,”IEEE Transactions on software engineering, vol. 39, no. 6, pp. 822–834, 2012

    2012

  8. [8]

    What is wrong with topic modeling? and how to fix it using search-based software engineering,

    A. Agrawal, W. Fu, and T. Menzies, “What is wrong with topic modeling? and how to fix it using search-based software engineering,” Information and Software Technology, vol. 98, pp. 74–88, 2018

    2018

Show all 63 references
  1. [9]

    On the time-based conclusion stability of cross-project defect prediction models,

    A. A. Bangash, H. Sahar, A. Hindle, and K. Ali, “On the time-based conclusion stability of cross-project defect prediction models,”Empirical Software Engineering, vol. 25, pp. 5047–5083, 2020

    2020

  2. [10]

    Shaky structures: The wobbly world of causal graphs in software analytics,

    J. Hulse, N. U. Eisty, and T. Menzies, “Shaky structures: The wobbly world of causal graphs in software analytics,”Empirical Software Engineering, vol. 30, no. 5, p. 142, 2025

    2025

  3. [11]

    Analyzing the instability of large language models in automated bug injection and correction,

    M. B. Er, N. ˙Ilhan, and U. Kuran, “Analyzing the instability of large language models in automated bug injection and correction,”arXiv preprint arXiv:2509.06429, 2025

    2025 arXiv

  4. [12]

    Can causality cure confusion caused by correlation (in software analytics)?

    A. Rayegan and T. Menzies, “Can causality cure confusion caused by correlation (in software analytics)?”arXiv preprint arXiv:2602.16091, 2026

    2026

  5. [13]

    Statistical modeling: The two cultures (with comments and a rejoinder by the author),

    L. Breiman, “Statistical modeling: The two cultures (with comments and a rejoinder by the author),”Statistical science, vol. 16, no. 3, pp. 199–231, 2001

    2001

  6. [14]

    Exploring the whole rashomon set of sparse decision trees,

    R. Xin, C. Zhong, Z. Chen, T. Takagi, M. Seltzer, and C. Rudin, “Exploring the whole rashomon set of sparse decision trees,”Advances in neural information processing systems, vol. 35, pp. 14 071–14 084, 2022

    2022

  7. [15]

    An empirical study of robustness and stability of machine learning classifiers in software defect prediction,

    A. Kaur and K. Kaur, “An empirical study of robustness and stability of machine learning classifiers in software defect prediction,” inAdvances in intelligent informatics. Springer, 2015, pp. 383–397

    2015

  8. [16]

    Bellwethers: A baseline method for transfer learning,

    R. Krishna and T. Menzies, “Bellwethers: A baseline method for transfer learning,”IEEE Transactions on Software Engineering, vol. 45, no. 11, pp. 1081–1105, 2018

    2018

  9. [17]

    Problems and opportunities in training deep learning software systems: An analysis of variance,

    H. V . Pham, S. Qian, J. Wang, T. Lutellier, J. Rosenthal, L. Tan, Y . Yu, and N. Nagappan, “Problems and opportunities in training deep learning software systems: An analysis of variance,” inProceedings of the 35th IEEE/ACM international conference on automated software engin...

    2020

  10. [18]

    On the reproducibility and replicability of deep learning in software engineering,

    C. Liu, C. Gao, X. Xia, D. Lo, J. Grundy, and X. Yang, “On the reproducibility and replicability of deep learning in software engineering,”ACM Trans. Softw. Eng. Methodol., 2021. [Online]. Available: https://doi.org/10.1145/3477535

    2021 doi

  11. [19]

    How to “dodge

    A. Agrawal, W. Fu, D. Chen, X. Shen, and T. Menzies, “How to “dodge” complex software analytics,”IEEE Transactions on Software Engineering, vol. 47, no. 10, pp. 2182–2194, 2019

    2019

  12. [20]

    The design, analysis and interpretation of repertory grids,

    M. Easterby-Smith, “The design, analysis and interpretation of repertory grids,”International Journal of Man-Machine Studies, vol. 13, no. 1, pp. 3–24, 1980

    1980

  13. [21]

    Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software,

    T. Xu, L. Jin, X. Fan, Y . Zhou, S. Pasupathy, and R. Talwadker, “Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software,” inProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, ...

    2015

  14. [22]

    The current challenges of software engineering in the era of large language models,

    C. Gao, X. Hu, S. Gao, X. Xia, and Z. Jin, “The current challenges of software engineering in the era of large language models,”ACM Transactions on Software Engineering and Methodology, vol. 34, no. 5, pp. 1–30, 2025

    2025

  15. [23]

    Prompting instability: An empirical study of llm robustness in code vulnerability detection,

    S. Han, T. Tan, Y . Miao, X. Chen, and N. Sun, “Prompting instability: An empirical study of llm robustness in code vulnerability detection,” inAustralasian Joint Conference on Artificial Intelligence. Springer, 2025, pp. 233–245

    2025

  16. [24]

    Reproducibility in machine-learning- based research: Overview, barriers, and drivers,

    H. Semmelrock, T. Ross-Hellauer, S. Kopeinik, D. Theiler, A. Haberl, S. Thalmann, and D. Kowald, “Reproducibility in machine-learning- based research: Overview, barriers, and drivers,”AI Magazine, vol. 46, no. 2, p. e70002, 2025

    2025

  17. [25]

    Oops!... i did it again. conclusion (in-) stability in quantitative empirical software engineering: A large-scale analysis,

    N. Hoess, C. Paradis, R. Kazman, and W. Mauerer, “Oops!... i did it again. conclusion (in-) stability in quantitative empirical software engineering: A large-scale analysis,”arXiv preprint arXiv:2510.06844, 2025

    2025

  18. [26]

    Tuning for software analytics: Is it really necessary?

    W. Fu, T. Menzies, and X. Shen, “Tuning for software analytics: Is it really necessary?”Information and Software Technology, vol. 76, pp. 135–146, 2016

    2016

  19. [27]

    An empirical comparison of validation methods for software prediction models,

    A. Ali and C. Gravino, “An empirical comparison of validation methods for software prediction models,”Journal of Software: Evolution and Process, vol. 33, no. 8, p. e2367, 2021

    2021

  20. [28]

    On the existence of simpler machine learning models,

    L. Semenova, C. Rudin, and R. Parr, “On the existence of simpler machine learning models,” inProceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, ser. FAccT ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 1827–1858. [Onli...

    2022 doi

  21. [29]

    Amazing things come from having many good models,

    C. Rudin, C. Zhong, L. Semenova, M. Seltzer, R. Parr, J. Liu, S. Katta, J. Donnelly, H. Chen, and Z. Boner, “Amazing things come from having many good models,”arXiv preprint arXiv:2407.04846, 2024

    2024 arXiv

  22. [30]

    A path to simpler models starts with noise,

    L. Semenova, H. Chen, R. Parr, and C. Rudin, “A path to simpler models starts with noise,”Advances in neural information processing systems, vol. 36, pp. 3362–3401, 2023

    2023

  23. [31]

    Data quality matters: A case study on data label correctness for security bug report prediction,

    X. Wu, W. Zheng, X. Xia, and D. Lo, “Data quality matters: A case study on data label correctness for security bug report prediction,”IEEE Transactions on Software Engineering, vol. 48, no. 7, pp. 2541–2556, 2021

    2021

  24. [32]

    An em- pirical study of the classification performance of learners on imbalanced and noisy software quality data,

    C. Seiffert, T. M. Khoshgoftaar, J. Van Hulse, and A. Folleco, “An em- pirical study of the classification performance of learners on imbalanced and noisy software quality data,” in2007 IEEE International Conference on Information Reuse and Integration, 2007, pp. 651–658

    2007

  25. [33]

    Finding faster configurations using flash,

    V . Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,”IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 794–811, 2020

    2020

  26. [34]

    Minimal data, maximum clarity: A heuristic for explaining optimization,

    A. Rayegan and T. Menzies, “Minimal data, maximum clarity: A heuristic for explaining optimization,”Journal of Systems and Software, vol. 238, p. 112897, 2026. [Online]. Available: https: //www.sciencedirect.com/science/article/pii/S0164121226001305 11

    2026

  27. [35]

    Can llms replace manual annotation of software engineering artifacts?

    T. Ahmed, P. Devanbu, C. Treude, and M. Pradel, “Can llms replace manual annotation of software engineering artifacts?” inMSR, 2025, pp. 526–538

    2025

  28. [36]

    Frugal: Unlocking semi-supervised learning for software analytics,

    H. Tu and T. Menzies, “Frugal: Unlocking semi-supervised learning for software analytics,” inASE, 2021, pp. 394–406

    2021

  29. [37]

    Heuristics for systems engineering cost estimation,

    R. Valerdi, “Heuristics for systems engineering cost estimation,”IEEE Systems Journal, vol. 5, no. 1, pp. 91–98, 2010

    2010

  30. [38]

    Identifying self-admitted technical debts with jitterbug: A two-step approach,

    Z. Yu, F. M. Fahid, H. Tu, and T. Menzies, “Identifying self-admitted technical debts with jitterbug: A two-step approach,”IEEE Trans. Softw. Eng., 2021

    2021

  31. [39]

    Detecting false alarms from automatic static analysis tools: How far are we?

    H. J. Kang, K. L. Aw, and D. Lo, “Detecting false alarms from automatic static analysis tools: How far are we?” in44th Int. Conf. Softw. Eng., 2022

    2022

  32. [40]

    Data quality: Some comments on the nasa software defect datasets,

    M. Shepperd, Q. Song, Z. Sun, and C. Mair, “Data quality: Some comments on the nasa software defect datasets,”IEEE Trans. Softw. Eng., vol. 39, no. 9, pp. 1208–1215, 2013

    2013

  33. [41]

    A large-scale empirical study of just-in-time quality assurance,

    Y . Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,”IEEE Trans. Softw. Eng., vol. 39, no. 6, pp. 757–773, 2012

    2012

  34. [42]

    Transferring performance prediction models across different hardware platforms,

    P. Valov, J. Petkovich, J. Guo, S. Fischmeister, and K. Czarnecki, “Transferring performance prediction models across different hardware platforms,” inProc. of the 8th ACM/SPEC on Int. Conf. Perf. Eng., 2017, pp. 39–50

    2017

  35. [43]

    Moot: a repository of many multi-objective optimiza- tion tasks,

    T. Menzies, T. Chen, Y . Ye, K. K. Ganguly, A. Rayegan, S. Srinivasan, and A. Lustosa, “Moot: a repository of many multi-objective optimiza- tion tasks,” in2026 IEEE/ACM 23rd International Conference on Mining Software Repositories (MSR), 2026

    2026

  36. [44]

    Some methods for classification and analysis of multi- variate observations,

    J. MacQueen, “Some methods for classification and analysis of multi- variate observations,” inProceedings of the 5th Berkeley Symposium on Mathematical Statistics and Probability, vol. 1. University of California Press, 1967, pp. 281–297

    1967

  37. [45]

    Density-based clustering based on hierarchical density estimates,

    R. J. G. B. Campello, D. Moulavi, and J. Sander, “Density-based clustering based on hierarchical density estimates,” inAdvances in Knowledge Discovery and Data Mining (PAKDD), ser. Lecture Notes in Computer Science, vol. 7819. Springer, 2013, pp. 160–172

    2013

  38. [46]

    Cure: An efficient clustering algorithm for large databases,

    S. Guha, R. Rastogi, and K. Shim, “Cure: An efficient clustering algorithm for large databases,”ACM SIGMOD Record, vol. 27, no. 2, pp. 73–84, 1998

    1998

  39. [47]

    Pearl,Causality

    J. Pearl,Causality. Cambridge university press, 2009

    2009

  40. [48]

    Pearl and D

    J. Pearl and D. Mackenzie,The Book of Why: The New Science of Cause and Effect, 1st ed. USA: Basic Books, Inc., 2018

    2018

  41. [49]

    Unicorn: Reasoning about configurable system performance through the lens of causality,

    M. S. Iqbal, R. Krishna, M. A. Javidian, B. Ray, and P. Jamshidi, “Unicorn: Reasoning about configurable system performance through the lens of causality,” ser. EuroSys ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 199–217. [Online]. Available: https:/...

    2022 doi

  42. [50]

    Promisetune: Unveiling causally promising and explainable configuration tuning,

    P. Chen and T. Chen, “Promisetune: Unveiling causally promising and explainable configuration tuning,” 2025

    2025

  43. [51]

    Assessing decision tree stability: a comprehensive method for generating a stable decision tree,

    J. Lee, M. K. Sim, and J.-S. Hong, “Assessing decision tree stability: a comprehensive method for generating a stable decision tree,”IEEE Access, vol. 12, pp. 90 061–90 072, 2024

    2024

  44. [52]

    Can ai be easy? lessons learned from the ezr.py toolkit,

    T. Menzies and S. Srinivasan, “Can ai be easy? lessons learned from the ezr.py toolkit,” 2026. [Online]. Available: https://arxiv.org/abs/2606. 03640

    2026

  45. [53]

    The case for compact ai,

    T. Menzies, “The case for compact ai,”Communications of the ACM, vol. 68, no. 8, pp. 6–7, 2025

    2025

  46. [54]

    How low can you go? the data-light se challenge,

    K. K. Ganguly and T. Menzies, “How low can you go? the data-light se challenge,”Proceedings of the ACM on Software Engineering, no. FSE, 2026

    2026

  47. [55]

    Zoom, don’t wander: Why regional search outperforms pareto reasoning and global optimization in budget- constrained sbse,

    K. Ganguly and T. Menzies, “Zoom, don’t wander: Why regional search outperforms pareto reasoning and global optimization in budget- constrained sbse,”arXiv preprint arXiv:2605.09658, 2026

    2026 arXiv

  48. [56]

    Smac3: A versatile bayesian optimization package for hyperparameter optimization,

    M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter, “Smac3: A versatile bayesian optimization package for hyperparameter optimization,”Jour- nal of Machine Learning Research, vol. 23, no. 54, pp. 1–9, 2022

    2022

  49. [57]

    Stability of feature selection algorithms: a study on high-dimensional spaces,

    A. Kalousis, J. Prados, and M. Hilario, “Stability of feature selection algorithms: a study on high-dimensional spaces,”Knowledge and infor- mation systems, vol. 12, no. 1, pp. 95–116, 2007

    2007

  50. [58]

    A systematic review of effect size in software engineering experiments,

    V . B. Kampenes, T. Dyb˚a, J. E. Hannay, and D. I. Sjøberg, “A systematic review of effect size in software engineering experiments,”Information and Software Technology, vol. 49, no. 11-12, pp. 1073–1086, 2007

    2007

  51. [59]

    On the kolmogorov-smirnov test for normality with mean and variance unknown,

    H. W. Lilliefors, “On the kolmogorov-smirnov test for normality with mean and variance unknown,”Journal of the American statistical Association, vol. 62, no. 318, pp. 399–402, 1967

    1967

  52. [60]

    Parametric measures of effect size,

    R. Rosenthal, H. Cooper, L. Hedgeset al., “Parametric measures of effect size,”The handbook of research synthesis, vol. 621, no. 2, pp. 231–244, 1994

    1994

  53. [61]

    New effect size rules of thumb,

    S. S. Sawilowsky, “New effect size rules of thumb,”Journal of modern applied statistical methods, vol. 8, no. 2, p. 26, 2009

    2009

  54. [62]

    Cliff’s delta calculator: A non-parametric effect size program for two groups of observations,

    G. Macbeth, E. Razumiejczyk, and R. D. Ledesma, “Cliff’s delta calculator: A non-parametric effect size program for two groups of observations,”Universitas Psychologica, vol. 10, no. 2, pp. 545–555, 2011

    2011

  55. [63]

    Heuristics of instability and stabilization in model selec- tion,

    L. Breiman, “Heuristics of instability and stabilization in model selec- tion,”The Annals of Statistics, vol. 24, no. 6, pp. 2350–2383, 1996. 12

    1996

Pith tools

Reviewed July 14, 2026 · model on record in the stance chip above.