pith. sign in

arxiv: 2509.08667 · v2 · submitted 2025-09-10 · 💻 cs.SE

Minimal Data, Maximum Clarity: A Heuristic for Explaining Optimization

Amirali Rayegan , Tim Menzies This is my paper

Pith reviewed 2026-05-18 17:43 UTC · model grok-4.3

classification 💻 cs.SE
keywords multi-objective optimizationactive learningexplainable AIdecision treessoftware configurationNaive Bayes samplinglabel efficiency
0
0 comments X

The pith

EZR uses active sampling on minimal data to reach over 90% of top optimization performance while producing clearer cohort explanations than LIME or SHAP.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces EZR as a lightweight pipeline that selects a small set of informative configurations through Naive Bayes active learning, builds models from those labels, and then renders the optimization logic as concise decision trees. This setup is meant to solve the practical problem in software engineering where full labeling of large configuration spaces is expensive and error-prone. The authors show through experiments on sixty datasets that the resulting models still capture most of the performance of exhaustive methods and that the tree-based rationales are easier to use than standard attribution techniques for both global patterns and individual decisions.

Core claim

The central claim is that a Maximum Clarity Heuristic makes it possible and often preferable to generate effective multi-objective optimization and transparent explanations from far fewer but better-chosen examples rather than from complete supervised data, with EZR reliably delivering over 90 percent of best-known performance across real-world software engineering datasets while supplying cohort-based rationales that practitioners find clearer than those from LIME, SHAP, or BreakDown.

What carries the argument

The EZR pipeline, which interleaves Naive Bayes active sampling to locate high-quality configurations and decision-tree distillation to convert optimization logic into concise, readable rationales for both global trends and local choices.

If this is right

  • Fewer labels suffice for competitive optimization results when sampling focuses on informative cases rather than uniform coverage.
  • Decision trees distilled from the sampled data supply both global and local rationales that users rate higher in clarity than attribution scores from LIME, SHAP, or BreakDown.
  • The same lightweight pipeline can be reused across many different software engineering optimization tasks without per-problem redesign.
  • The endorsement of 'less but better' data directly reduces the labeling burden that currently limits practical deployment of optimization tools.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same sampling-plus-tree pattern could be tested on configuration problems outside software engineering, such as hardware or cloud resource tuning.
  • If the decision trees remain compact on higher-dimensional spaces, they might serve as a lightweight surrogate model for rapid what-if analysis during iterative design.
  • Combining the active sampler with other base learners besides Naive Bayes might improve performance on datasets where the current assumption of feature independence does not hold.

Load-bearing premise

Naive Bayes active sampling will reliably locate high-quality configurations on diverse software engineering datasets without needing extra tuning or suffering from mismatch with the underlying distribution.

What would settle it

A fresh collection of software configuration datasets on which EZR either falls below 90 percent of the best-known performance or receives lower clarity ratings from practitioners than the same tasks explained by SHAP or LIME.

Figures

Figures reproduced from arXiv: 2509.08667 by Amirali Rayegan, Tim Menzies.

Figure 1
Figure 1. Figure 1: EZR generates a very small tree summarizing the major points of the data. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: COC1000 Feature importance via permutation [PITH_FULL_IMAGE:figures/full_fig_p024_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: EZR tree output showing impurity-based splits and performance metrics at [PITH_FULL_IMAGE:figures/full_fig_p025_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: LIME explanation output Moving on to SHAP, [PITH_FULL_IMAGE:figures/full_fig_p026_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: SHAP explanation output Impurity), enabling direct inspection of how comparable cases performed in practice. It is important to note that this path involves only six features, while the full EZR tree uses nine drawn from seventeen. This reduction underscores EZR’s ability to highlight only the most informative attributes, which aligns with our Maximum Clarity Heuristic that complex tasks can often be expla… view at source ↗
Figure 6
Figure 6. Figure 6: BreakDown explanation output [PITH_FULL_IMAGE:figures/full_fig_p027_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Highlighted decision path in the EZR tree [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: In what portion of each dataset group does each method achieve 75% of the [PITH_FULL_IMAGE:figures/full_fig_p034_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: In what portion of each dataset group does each method achieve 90% of the [PITH_FULL_IMAGE:figures/full_fig_p034_9.png] view at source ↗
read the original abstract

Efficient, interpretable optimization is a critical but underexplored challenge in software engineering, where practitioners routinely face vast configuration spaces and costly, error-prone labeling processes. This paper introduces EZR, a novel and modular framework for multi-objective optimization that unifies active sampling, learning, and explanation within a single, lightweight pipeline. Departing from conventional wisdom, our Maximum Clarity Heuristic demonstrates that using less (but more informative) data can yield optimization models that are both effective and deeply understandable. EZR employs an active learning strategy based on Naive Bayes sampling to efficiently identify high-quality configurations with a fraction of the labels required by fully supervised approaches. It then distills optimization logic into concise decision trees, offering transparent, actionable explanations for both global and local decision-making. Extensive experiments across 60 real-world datasets establish that EZR reliably achieves over 90% of the best-known optimization performance in most cases, while providing clear, cohort-based rationales that surpass standard attribution-based explainable AI (XAI) methods (LIME, SHAP, BreakDown) in clarity and utility. These results endorse "less but better"; it is both possible and often preferable to use fewer (but more informative) examples to generate label-efficient optimization and explanations in software systems. To support transparency and reproducibility, all code and experimental materials are publicly available at https://github.com/amiiralii/Minimal-Data-Maximum-Clarity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces EZR, a modular framework for multi-objective optimization in software engineering. It combines Naive Bayes active sampling to identify high-quality configurations using minimal labels, followed by distillation into concise decision trees that provide global and local cohort-based explanations. Experiments on 60 real-world datasets claim that EZR reaches over 90% of best-known optimization performance while delivering clearer, more actionable rationales than attribution-based XAI baselines (LIME, SHAP, BreakDown), endorsing a 'less but better' data heuristic. All code and materials are released publicly.

Significance. If the central performance and clarity claims hold after addressing sampling assumptions, the work would offer a practical, label-efficient alternative to black-box optimization and post-hoc explanation methods in SE configuration tuning. The emphasis on minimal informative data and the public release of code and experimental materials are notable strengths that support reproducibility and adoption. The results, if robust, could shift practice toward inherently interpretable pipelines rather than separate optimization and explanation stages.

major comments (2)
  1. [§3] §3 (Active Sampling and Naive Bayes): The headline claim of reliably achieving ≥90% of best-known performance depends on the active sampler surfacing high-quality points. The method factors P(x) = ∏ P(x_i) and therefore cannot model parameter interactions common in SE configuration spaces (e.g., compiler flags or hyper-parameter dependencies). No ablation replacing Naive Bayes with an interaction-aware sampler (e.g., random forest or Gaussian-process-based) is reported, leaving the performance numbers unsupported when interactions dominate.
  2. [Experimental Evaluation] Experimental Evaluation (results across 60 datasets): The comparisons to LIME, SHAP, and BreakDown report superior clarity and utility, yet provide no details on baseline implementations, hyper-parameter settings, statistical significance tests, or controls for data leakage. Without these, it is impossible to verify whether the clarity advantage is robust or an artifact of particular choices.
minor comments (2)
  1. [Abstract and §1] The abstract and introduction repeatedly use 'cohort-based rationales' without a precise definition or example until later sections; a brief forward reference would improve readability.
  2. [Figures] Figure captions for the decision-tree visualizations could explicitly state the dataset and objective being illustrated to aid quick interpretation.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive and detailed review. We address the two major comments point by point below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [§3] §3 (Active Sampling and Naive Bayes): The headline claim of reliably achieving ≥90% of best-known performance depends on the active sampler surfacing high-quality points. The method factors P(x) = ∏ P(x_i) and therefore cannot model parameter interactions common in SE configuration spaces (e.g., compiler flags or hyper-parameter dependencies). No ablation replacing Naive Bayes with an interaction-aware sampler (e.g., random forest or Gaussian-process-based) is reported, leaving the performance numbers unsupported when interactions dominate.

    Authors: We agree that the independence assumption in Naive Bayes limits its ability to capture parameter interactions. At the same time, the reported results across 60 real-world datasets show that the active-sampling heuristic still reaches over 90 % of best-known performance in the great majority of cases. This empirical outcome is what underpins the 'less but better' claim. To directly address the referee's concern, the revised manuscript will add an ablation that substitutes the Naive Bayes sampler with an interaction-aware model (random forest) and reports the resulting performance delta. revision: yes

  2. Referee: [Experimental Evaluation] Experimental Evaluation (results across 60 datasets): The comparisons to LIME, SHAP, and BreakDown report superior clarity and utility, yet provide no details on baseline implementations, hyper-parameter settings, statistical significance tests, or controls for data leakage. Without these, it is impossible to verify whether the clarity advantage is robust or an artifact of particular choices.

    Authors: We appreciate the need for full reproducibility details. The revised Experimental Evaluation section will specify the exact library versions and hyper-parameter settings used for LIME, SHAP, and BreakDown; describe the statistical tests performed (Wilcoxon signed-rank tests with reported p-values and effect sizes); and clarify that all active sampling, model training, and explanation generation were conducted independently within each of the 60 datasets, eliminating cross-dataset leakage. revision: yes

Circularity Check

0 steps flagged

No significant circularity; claims rest on external datasets and standard baselines

full rationale

The paper describes EZR as an active-learning pipeline (Naive Bayes sampling to select configurations, followed by decision-tree distillation for explanations) and reports empirical results on 60 public real-world SE datasets, measuring performance against independently established best-known optimization outcomes and comparing explanations to LIME/SHAP/BreakDown. No derivation step equates the reported ≥90% performance or clarity metrics to a fitted parameter, self-citation, or input by construction; the sampling and tree steps are presented as procedural choices whose outputs are then evaluated on held-out or external benchmarks. The method is self-contained against those external references, with code released for reproduction.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based on abstract only; the framework assumes standard active-learning assumptions hold for software configuration spaces and that decision trees can faithfully distill the optimization logic without significant loss of fidelity. No explicit free parameters or invented entities are named in the abstract.

axioms (1)
  • domain assumption Naive Bayes provides a sufficiently accurate surrogate for guiding active sampling in multi-objective software configuration spaces
    Invoked by the choice of active learning strategy described in the abstract.

pith-pipeline@v0.9.0 · 5783 in / 1268 out tokens · 34737 ms · 2026-05-18T17:43:12.337657+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Zoom, Don't Wander: Why Regional Search Outperforms Pareto Reasoning and Global Optimization in Budget-Constrained SBSE

    cs.SE 2026-05 unverdicted novelty 4.0

    A minimal greedy regional zoom method outperforms Pareto and global Bayesian optimization in budget-constrained SBSE, winning or tying in 84-89% of cases at equal budget and even at one-fifth budget, because optimal s...

Reference graph

Works this paper leans on

55 extracted references · 55 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    T. Xu, X. Jin, P. Huang, Y. Zhou, S. Lu, L. Jin, S. Pasupathy, Early detection of configuration errors to reduce failure damage, in: Proc. OSDI 2016, 2016

    work page 2016

  2. [2]

    T. Chen, M. Li, Adapting multi-objectivized software configuration tuning, Proc. ACM Softw. Eng. (2024)

    work page 2024

  3. [3]

    Sayagh, N

    M. Sayagh, N. Kerzazi, B. Adams, F. Petrillo, Software configuration engi- neering in practice interviews, survey, and systematic literature review, IEEE Trans. Softw. Eng. (2018)

    work page 2018

  4. [4]

    T. Xu, L. Jin, X. Fan, Y. Zhou, S. Pasupathy, R. Talwadker, Hey, you have given me too many knobs!: understanding and dealing with over-designed configuration in system software, in: Proc. FSE 2015, 2015

    work page 2015

  5. [5]

    Mühlbauer, F

    S. Mühlbauer, F. Sattler, C. Kaltenecker, J. Dorn, S. Apel, N. Siegmund, Analysing the impact of workloads on modeling the performance of config- urable software systems, in: Proc. ICSE 2023, 2023

    work page 2023

  6. [6]

    co-training

    S. Majumder, J. Chakraborty, T. Menzies, When less is more: on the value of “co-training” for semi-supervised software defect predictors, Empir. Softw. Eng. (2024)

    work page 2024

  7. [7]

    J. A. Pereira, M. Acher, H. Martin, J.-M. Jézéquel, G. Botterweck, A. Ven- tresque, Learning software configuration spaces: A systematic literature re- view, J. Syst. Softw. (2021)

    work page 2021

  8. [8]

    X. Wu, W. Zheng, X. Xia, D. Lo, Data quality matters: A case study on data label correctness for security bug report prediction, IEEE Trans. Softw. Eng. 48 (2021)

    work page 2021

  9. [9]

    A. K. Shakya, G. Pillai, S. Chakrabarty, Reinforcement learning algorithms: A brief survey, Expert Syst. with Appl. 231 (2023)

    work page 2023

  10. [10]

    Bergstra, Y

    J. Bergstra, Y. Bengio, Random search for hyper-parameter optimization, The journal of machine learning research (2012)

    work page 2012

  11. [11]

    Z. Li, M. Harman, R. M. Hierons, Search algorithms for regression test case prioritization, IEEE Trans. Softw. Eng. (2007)

    work page 2007

  12. [12]

    Khatibsyarbini, M

    M. Khatibsyarbini, M. A. Isa, D. N. Jawawi, R. Tumeng, Test case prioriti- zation approaches in regression testing: A systematic literature review, Inf. Softw. Technol. (2018)

    work page 2018

  13. [13]

    Hujainah, R

    F. Hujainah, R. B. A. Bakar, M. A. Abdulgabber, K. Z. Zamli, Software re- quirements prioritisation: a systematic literature review on significance, stake- holders, techniques and challenges, IEEE Access (2018)

    work page 2018

  14. [14]

    Zhang, M

    Y. Zhang, M. Harman, S. A. Mansouri, The multi-objective next release prob- lem, in: Proc. GECCO 2007, 2007

    work page 2007

  15. [15]

    A. V. Rezende, L. Silva, A. Britto, R. Amaral, Software project scheduling problem in the context of search-based software engineering: A systematic review, J. Syst. Softw. (2019). 40

    work page 2019

  16. [16]

    Sarro, Search-based software engineering in the era of modern software systems, in: Proc

    F. Sarro, Search-based software engineering in the era of modern software systems, in: Proc. IEEE Int. Conf. on Requirements Engineering, 2023

    work page 2023

  17. [17]

    Mkaouer, M

    W. Mkaouer, M. Kessentini, A. Shaout, P. Koligheu, S. Bechikh, K. Deb, A. Ouni, Many-objective software remodularization using nsga-iii, ACM TOSEM (2015)

    work page 2015

  18. [18]

    V. Nair, Z. Yu, T. Menzies, N. Siegmund, S. Apel, Finding faster configura- tions using flash, IEEE Trans. Softw. Eng. (2018)

    work page 2018

  19. [19]

    M. Li, T. Chen, X. Yao, How to evaluate solutions in pareto-based search- based software engineering: A critical review and methodological guidance, IEEE Trans. Softw. Eng. (2020)

    work page 2020

  20. [20]

    Lustosa, T

    A. Lustosa, T. Menzies, Less noise, more signal: Drr for better optimizations of se tasks, arxiv:2503.21086 (2025)

    work page arXiv 2025

  21. [21]

    B. W. Boehm, Software engineering economics, in: Software pioneers: Con- tributions to Softw. Eng., Springer, 2011

    work page 2011

  22. [22]

    A. H. Mohammadkhani, N. S. Bommi, M. Daboussi, O. Sabnis, C. Tan- tithamthavorn, H. Hemmati, A systematic literature review of explainable ai for software engineering, arxiv:2302.06065 (2023)

    work page arXiv 2023

  23. [23]

    Arora, S

    L. Arora, S. S. Girija, S. Kapoor, A. Raj, D. Pradhan, A. Shetgaonkar, Ex- plainable artificial intelligence techniques for software development lifecycle: A phase-specific survey, arxiv:2505.07058 (2025)

    work page arXiv 2025

  24. [24]

    Tantithamthavorn, J

    C. Tantithamthavorn, J. Cito, H. Hemmati, S. Chandra, Explainable ai for se: Challenges and future directions, IEEE Softw. 40 (2023)

    work page 2023

  25. [25]

    C. K. Tantithamthavorn, J. Jiarpakdee, Explainable ai for software engineer- ing, in: Proc. ASE 2021, 2021

    work page 2021

  26. [26]

    Dwivedi, D

    R. Dwivedi, D. Dave, H. Naik, S. Singhal, R. Omer, P. Patel, B. Qian, Z. Wen, T. Shah, G. Morgan, R. Ranjan, Explainable ai (xai): Core ideas, techniques, and solutions, ACM Comput. Surv. (2023)

    work page 2023

  27. [27]

    Biecek, T

    P. Biecek, T. Burzykowski, Explanatory model analysis: explore, explain, and examine predictive models, Chapman and Hall/CRC, 2021

    work page 2021

  28. [28]

    why should i trust you?

    M. T. Ribeiro, S. Singh, C. Guestrin, " why should i trust you?" explaining the predictions of any classifier, in: Proc. KDD 2016, 2016

    work page 2016

  29. [29]

    S. M. Lundberg, S.-I. Lee, A unified approach to interpreting model predic- tions, Adv. Neural Inf. Process. Syst. (2017)

    work page 2017

  30. [30]

    Staniak, P

    M. Staniak, P. Biecek, Explanations of model predictions with live and break- down packages, The R Journal (2018)

    work page 2018

  31. [31]

    D. Chen, W. Fu, R. Krishna, T. Menzies, Applications of psychological science for actionable analytics, in: Proc. FSE, 2018

    work page 2018

  32. [32]

    G. A. Miller, The magical number seven, plus or minus two: Some limits on our capacity for processing information., Psychol. Rev. (1956). 41

    work page 1956

  33. [33]

    R. R. Hoffman, S. T. Mueller, G. Klein, M. Jalaeian, C. Tate, Explainable ai: roles and stakeholders, desirements and challenges, Frontiers in Computer Science (2023)

    work page 2023

  34. [34]

    right to explanation

    B. Goodman, S. Flaxman, European union regulations on algorithmic decision making and a “right to explanation”, AI Mag. (2017)

    work page 2017

  35. [35]

    Barredo Arrieta, N

    A. Barredo Arrieta, N. Díaz-Rodríguez, J. Del Ser, A. Bennetot, S. Tabik, A. Barbado, S. Garcia, S. Gil-Lopez, D. Molina, R. Benjamins, R. Chatila, F. Herrera, Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai, Information Fusion (2020)

    work page 2020

  36. [36]

    Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance.arXiv preprint arXiv:2304.11127, 2023

    S. Watanabe, Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance, arxiv:2304.11127 (2023)

    work page arXiv 2023

  37. [37]

    Breiman, Random forests, Machine learning (2001)

    L. Breiman, Random forests, Machine learning (2001)

    work page 2001

  38. [38]

    Arreche, T

    O. Arreche, T. Guntur, M. Abdallah, Xai-based feature selection for improved network intrusion detection systems, arxiv:2410.10050 (2024)

    work page arXiv 2024

  39. [39]

    Hinterleitner, T

    A. Hinterleitner, T. Bartz-Beielstein, R. Schulz, S. Spengler, T. Winter, C. Leitenmeier, Enhancing feature selection and interpretability in ai regres- sion tasks through feature attribution, arxiv:2409.16787 (2024)

    work page arXiv 2024

  40. [40]

    W. E. Marcílio, D. M. Eler, From explanations to feature selection: assessing shap values as feature selection mechanism, in: Proc. SIBGRAPI 2020, 2020

    work page 2020

  41. [41]

    Robnik-Šikonja, I

    M. Robnik-Šikonja, I. Kononenko, et al., An adaptation of relief for attribute estimation in regression, in: Machine learning: Proceedings of the fourteenth international conference (ICML’97), 1997

    work page 1997

  42. [42]

    L. St, S. Wold, et al., Analysis of variance (anova), Chemometrics and intel- ligent laboratory systems (1989)

    work page 1989

  43. [43]

    A. J. Scott, M. Knott, A cluster analysis method for grouping means in the analysis of variance, Biometrics (1974)

    work page 1974

  44. [44]

    Senthilkumar, T

    L. Senthilkumar, T. Menzies, Can large language models improve se active learning via warm-starts? (2024)

    work page 2024

  45. [45]

    K. K. Ganguly, T. Menzies, Bingo! simple optimizers win big if problems collapse to a few buckets (2025)

    work page 2025

  46. [46]

    Menzies, T

    T. Menzies, T. Chen, Moot repository of multi-objective optimization tests (2025). URLhttp://github.com/timm/moot

    work page 2025

  47. [47]

    Amershi, A

    S. Amershi, A. Begel, C. Bird, R. DeLine, H. Gall, E. Kamar, N. Nagappan, B. Nushi, T. Zimmermann, Software engineering for machine learning: A case study, in: Proc. ICSE-SEIP 2019, 2019

    work page 2019

  48. [48]

    F. J. Alcaide, J. R. Romero, A. Ramírez, Can explainable artificial intelligence support software modelers in model comprehension?, Software and Syst. Mod- eling (2025). 42

    work page 2025

  49. [49]

    R. R. Hoffman, S. T. Mueller, G. Klein, J. Litman, Metrics for explainable ai: Challenges and prospects, arxiv:1812.04608 (2018)

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  50. [50]

    Pearl, D

    J. Pearl, D. Mackenzie, The book of why: the new science of cause and effect, Basic books, 2018

    work page 2018

  51. [51]

    D.N.Palacio, A.Velasco, N.Cooper, A.Rodriguez, K.Moran, D.Poshyvanyk, Toward a theory of causation for interpreting neural code models, IEEE Trans. Softw. Eng. (2024)

    work page 2024

  52. [52]

    Lustosa, T

    A. Lustosa, T. Menzies, isneak: Partial ordering as heuristics for model-based reasoning in software engineering, IEEE Access (2024)

    work page 2024

  53. [53]

    R. G. Hamlet, Probable correctness theory, Information processing letters (1987)

    work page 1987

  54. [54]

    C. A. Hoare, Algorithm 65: find, Communications of the ACM (1961)

    work page 1961

  55. [55]

    Giagkiozis, P

    I. Giagkiozis, P. Fleming, Methods for multi-objective optimization: An anal- ysis, Inf. Sci. (2015). 43

    work page 2015