Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

BINGO! Simple Optimizers Win Big if Problems Collapse to a Few Buckets

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Real software-engineering data collapses into a tiny fraction of possible buckets, so simple sampling can match expensive optimizers at a 10,000x speedup.

desk verdict A genuinely interesting bucket-occupancy measurement is undermined by an unfair DEHB comparison; the paper deserves revision, not rejection. read the letter →

arxiv 2506.04509 v1 pith:PQPK5WDC submitted 2025-06-04 cs.SE

Kishan Kumar Ganguly , Tim Menzies This is my paper
classification cs.SE
keywords BINGOeffectsearch-basedsoftwareengineeringmulti-objectiveoptimizationactivelearninghyperparameterconfigurationlabel-efficientsamplingdatapruning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Software-engineering optimization usually assumes that finding good configurations requires evaluating thousands to millions of candidates. This paper tries to establish a countervailing phenomenon, the BINGO effect: when data is divided into b bins along d dimensions, the number of buckets that actually contain data is far smaller than b^d, in their data roughly 100 filled buckets out of thousands possible. The authors argue that when this collapse holds, most search effort is wasted, and simple label-efficient samplers that stay inside the filled buckets can match or beat the state-of-the-art DEHB optimizer. Across 39 SE optimization tasks they show the effect, report that RANDOM runs 10,000x faster than DEHB while matching it, and introduce LITE and LINE for tight label budgets. The practical message is to test for bucket collapse before spending CPU on heavy optimization.

What carries the argument

The load-bearing mechanism is the bucket-count comparison. Rows are divided into buckets using Fastmap projections onto directions defined by distant corner rows, then each dimension is split into b bins; a bucket counts only if it has at least 2^d rows, following a DBSCAN-inspired density rule. The paper measures success by 'distance to heaven,' the normalized distance from a found row to the ideal objective point, summarized as Δ. Against that yardstick, LITE sorts labeled data into $\sqrt$(N) good and N - $\sqrt$(N) rest to build a two-class classifier that selects the next row maximizing good-to-rest likelihood, while LINE seeds centroids with K-Means++ for diversity; DEHB's generated rows are labeled by nearest neighbor in the dataset, following a benchmark procedure from prior work.

What would settle it

Run one of the smaller datasets with true evaluation of every DEHB candidate (feasible when rows are few) and compare against RANDOM, LITE, and LINE; if true-labeled DEHB ranks above the simple samplers, the 10,000x speedup claim loses its support. As a complement, find an SE optimization dataset whose data fills most of the b^d buckets; the paper predicts simple methods will not match DEHB there, so a counterexample would bound the BINGO effect.

Watch

Extended reading notes

Core claim

The central claim is that the BINGO effect, not algorithm engineering, explains why simple optimizers succeed in software engineering. The paper defines the effect by counting buckets: rows are projected onto a few dimensions using a Fastmap-style method, split into b bins per dimension, and a bucket is 'filled' only if it holds at least 2^d rows. In all 39 studied datasets, the filled-bucket count stays far below b^d, and increasing the number of bins or dimensions often does not increase the number of filled buckets. Exploiting this, the paper shows that pool-based samplers: RANDOM, LITE (a simplification of tree-of-Parzen-estimator active learning), and LINE (K-Means++ centroid selection), perform as well as or better than DEHB, a membership-query optimizer that generates new candidate rows. DEHB is the slowest and, on these datasets, the worst-ranked treatment; RANDOM is 10,000x faster and LITE/LINE are about 100x faster, with the best method shifting from LITE at budgets of 24 or fewer labels to LINE above that.

Load-bearing premise

The result depends on trusting that labeling a DEHB-generated configuration with the score of its nearest already-labeled row faithfully represents what actually evaluating that configuration would give, and that the 39 datasets stand in for software-engineering optimization more broadly.

Editorial extensions

If this is right

  • For any SE optimization task whose data exhibits the BINGO effect, evaluating thousands or millions of configurations is unnecessary; a few dozen labeled samples can reach over 90% of the attainable improvement.
  • DEHB's membership-query search outside the filled buckets gives no advantage over pool-based sampling on these 39 tasks, so CPU-heavy mutation-based optimization can be replaced by lightweight sampling when bucket collapse is present.
  • The best tool depends on the label budget: LITE wins with 24 or fewer labels, LINE wins above that, and performance plateaus after about 50 labels, suggesting a 'double BINGO' where only a subset of buckets needs exploration.
  • Practitioners can check for BINGO using only the independent x-values, without expensive labeling, and can avoid simple methods when data spreads out into many thousands of buckets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension the paper leaves implicit: the same collapse argument should apply beyond configuration tables, for instance to test-case generation and regression suites; if those domains also bucket-collapse, label-efficient sampling could shrink test suites, but that is not established by the 39 datasets used here.
  • The nearest-neighbor labeling of DEHB's novel rows is a surrogate for true evaluation; we infer that a true-label comparison on a few small datasets could shift DEHB's ranking, though the paper defends the procedure as standard benchmark practice.
  • The BINGO bucket count looks like a cheap, label-free estimator of a dataset's effective dimensionality; connecting n to intrinsic-dimension measures could predict when simple optimization works without running any optimizer at all.
  • The plateau after 50 labels suggests an automatic stopping rule: stop sampling when doubling the budget no longer improves the objective, which the paper's budget-doubling experiments already hint at.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces the BINGO effect, the observation that in 39 SE optimization datasets, data rows occupy far fewer buckets than the b^d possible cells after binning. It argues that this collapse explains why simple pool-based stochastic optimizers (RANDOM, LITE, LINE) match or beat DEHB, a state-of-the-art evolutionary hyperparameter optimizer, while being orders of magnitude faster. The paper poses four research questions: RQ1 (prevalence of BINGO), RQ2 (effectiveness of BINGO-based reasoning), RQ3 (speed), and RQ4 (simplicity). It reports bucket-count experiments (Algorithm 1, Figure 4), an optimizer comparison using distance-to-heaven and Scott-Knott ranking (Table V, Figure 5), and runtime comparisons (Figure 6). The principal positive result is that on these datasets simple methods often rank first, especially at small label budgets, and that the code and data are publicly released.

Significance. If the central claims hold, the paper provides a practically important and surprising result for SE optimization: many configuration and HPO problems are effectively low-dimensional, so expensive search can be replaced by cheap labeling. The strengths are the relatively large corpus of 39 datasets, the use of a state-of-the-art baseline (DEHB), the explicit budget-controlled label-cost framing, and the public artifact. The paper also correctly identifies a potential limitation in its own threats-to-validity section, namely sampling bias. However, the significance is currently bounded by two gaps: the BINGO explanation is not causally tied to the optimizer results, and the DEHB comparison uses a nearest-neighbor surrogate that handicaps DEHB. If the authors address these, the paper would be a useful empirical contribution to search-based SE and active learning.

major comments (3)
  1. [Section IV-B.2 and Section V (Table V, Figure 5)] The comparison of DEHB against LITE/LINE/RANDOM is unfair by construction. DEHB generates configurations outside the pool, but each generated row is labeled with the y-value of its nearest neighbor in Table I. This projects DEHB's search back onto the existing pool and guarantees that no out-of-pool configuration can receive a label better than the best nearby pool row. Meanwhile, the pool-based methods receive exact labels for the rows they select. The defense in Section IV-B.2 that "some form of nearest-neighbor testing is unavoidable" conflates the benchmark table with the true objective: the table is finite ground truth, but DEHB's membership queries are not evaluated against a real implementation of f. As written, the results measure a handicapped DEHB, not DEHB as an optimizer, and this affects RQ2, RQ3, and the abstract's parity and speedup claims. The authors should either evaluate DEHB on actual executions (or an independent validated surrogate) or explicitly restrict the claims to pool-based optimization and remove the DEHB parity claim.
  2. [Section III (Algorithm 1) and Section V] The paper does not test the claimed explanatory link between the BINGO effect and optimizer performance. Algorithm 1 counts filled buckets using x-values alone, while Table V reports the relative performance of the optimizers, but no analysis correlates per-dataset bucket counts with per-dataset optimizer rankings. The statement that simple methods succeed "because" data collapses to a few buckets is an inference, not a demonstrated result. A direct test would compute, for each of the 39 datasets, a bucket-count measure under a fixed protocol and then check whether datasets with fewer buckets show larger gains for RANDOM/LITE/LINE relative to DEHB. Without such an analysis, the BINGO effect remains a descriptive observation rather than an explanation.
  3. [Section I, contribution 7, and Section VI] The proposed guideline for when not to use simple methods is not validated. The introduction promises a "test for when to avoid simpler methods" (do not use them if data has too many buckets), and Section VI-B speculates about a "double BINGO effect," but no experiment identifies a bucket-count threshold, nor does the paper present any dataset where simple methods fail because of too many buckets. Since this guidance is listed as a contribution and is central to the paper's practical message, the authors should either provide evidence for a threshold or remove this claim.
minor comments (5)
  1. [Section III, Algorithm 1] The minimum bucket occupancy condition in line 10 is written as len(x) >= 2d, but the surrounding text says "filled buckets need at least 2d minPts," which could be read as 2^d. Please clarify whether the threshold is 2*d or 2^d, and ensure the pseudocode matches the text.
  2. [Section III, Figure 4] The text refers to "Figure III" and later "Algorithm IV," but the actual numbering uses Arabic numerals (Figure 4, Algorithm 1). These cross-references should be corrected.
  3. [Section III, RQ1 answer] The sentence "Empirically, it cannot be shown that this method applies to all data" is awkward and seems to undercut the stated RQ1 result. Consider rephrasing to state more clearly that the claim is restricted to the 39 studied datasets.
  4. [Section IV-C, Equation (2)] The definition of Delta uses x, min, and mu without explicitly stating that these are computed after normalization of the y-values. Please state this more explicitly to avoid confusion.
  5. [Section V, Table V] The table contains many numeric columns and color codings, but the legend for the color ranks is only in the caption; a reader may have difficulty matching cells to ranks. Consider adding a small inline legend or a separate column with the rank number.

Circularity Check

1 steps flagged · score 6.0 of 10

The headline parity with DEHB is forced by the nearest-neighbor labeling protocol, which maps every out-of-pool query back onto the pool.

  1. fitted input called prediction [Section IV-B.2 (Labeling), Experimental Rig]
    "However, for membership query inference tools such as DEHB, labeling is more complex since DEHB can generate new rows. This means that DEHB has to assign values to goals for previously unseen examples. Following the procedure outlined by Pfisterer and Zela et al. [61], [62], new rows are labeled by finding their nearest neighbor in Table I."

    This evaluation protocol defines DEHB's objective value for every generated configuration as the y-value of the nearest existing pool row. Therefore DEHB's search landscape is a piecewise-constant Voronoi projection of the pool, and its best attainable score is exactly the best pool row's score. Pool-based methods (RANDOM, LINE, LITE) query exact y-values from that same pool, so they can achieve that same best score. The paper's conclusion that simple stochastic selection 'can match complex optimizers like DEHB' is thus a mathematical consequence of the labeling rule, not an empirical discovery about optimizer strength.

full rationale

The BINGO effect itself is not circular: bucket counts in Algorithm 1 and Figure 4 are computed from x-values alone, and RQ1 is a descriptive empirical claim independent of the optimizer comparison. The central problem is in the optimizer evaluation. Section IV-B.2 labels every DEHB-generated row by the y-value of its nearest neighbor in the pool, so DEHB can never receive credit for an out-of-pool solution better than the best row in the pool. Since LITE, LINE, and RANDOM obtain exact labels for pool rows, they can attain that same best value, making the observed 'simple methods match DEHB' result an artifact of the evaluation design rather than a test of DEHB's search capability. This affects RQ2, RQ3, and the abstract's claim that simple stochastic selection matches complex optimizers. There are also many self-citations, but they are not load-bearing for the BINGO effect; the cited prior work is used for context and baselines, not to establish the paper's central phenomenon. Thus the paper is only partially circular: the BINGO measurement is self-contained, but a headline result reduces to the labeling construction.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central claims rest on a small number of ad hoc experimental choices and domain assumptions. The free parameters are the random bin and dimension ranges, the minimum bucket occupancy threshold, the label budgets, and the LITE good/rest split. The main domain assumptions are representativeness of the MOOT corpus, validity of Fastmap-derived buckets, adequacy of nearest-neighbor labels for DEHB, and suitability of distance-to-heaven as an evaluation measure. The only invented entity without independent support is the conjectured double BINGO effect.

free parameters (4)
  • Algorithm 1 dimension and bin ranges (d1, d2, b1, b2) = d1 in [3,8], d2 in [d1,8], b1 in [3,10], b2 in [b1,10]
    Chosen by hand to define the bucket survey; no sensitivity analysis is reported, yet the RQ1 bucket counts come from these ranges.
  • Minimum bucket occupancy threshold = 2*d rows (DBSCAN minPts heuristic)
    Line 10 of Algorithm 1 excludes sparse buckets; changing this threshold changes the reported BINGO counts.
  • Label budgets = {6, 12, 18, 24, 50, 100, 200}
    Experiment design choice; the LITE for <=24 labels and LINE for <=50 labels recommendations are read off these budgets.
  • LITE good/rest split = sqrt(N) good versus N - sqrt(N) rest
    Ad hoc simplification of TPE; no analysis of sensitivity to this split is provided.
assumptions (4)
  • domain assumption The 39 MOOT datasets are representative of SE optimization problems
    All datasets come from the authors' own MOOT repository, and the authors admit in Section VI-A that test case generation and reduction data are absent.
  • domain assumption Fastmap projections with randomly chosen corners reveal the true bucket structure of the data
    Invoked in Table IV and Algorithm 1; if the projections do not reflect the data's true low-dimensional structure, the bucket collapse could be an artifact of the binning procedure.
  • domain assumption Nearest-neighbor labeling is an adequate oracle for evaluating DEHB
    Used in Section IV-B.2 for membership query inference; this can bias the comparison for or against DEHB.
  • domain assumption Distance to heaven is a sufficient multi-objective performance measure
    Defined in Section IV-C; the authors checked one Chebyshev variant but did not use standard measures such as hypervolume.
invented entities (1)
  • Double BINGO effect
    purpose: Explains the performance plateau after 50 labels and motivates speculation that downstream benefits would arrive almost instantly
    Conjectured in Section V with no test or measurement; it is an untested post hoc explanation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of BINGO! Simple Optimizers Win Big if Problems Collapse to a Few Buckets." pith.science (2026). https://pith.science/paper/PQPK5WDC

@misc{pith2026250604509,
  author       = {Pith},
  title        = {Pith review of: BINGO! Simple Optimizers Win Big if Problems Collapse to a Few Buckets},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PQPK5WDC}},
  note         = {Machine review of arXiv:2506.04509}
}
read the original abstract

Traditional multi-objective optimization in software engineering (SE) can be slow and complex. This paper introduces the BINGO effect: a novel phenomenon where SE data surprisingly collapses into a tiny fraction of possible solution "buckets" (e.g., only 100 used from 4,096 expected). We show the BINGO effect's prevalence across 39 optimization in SE problems. Exploiting this, we optimize 10,000 times faster than state-of-the-art methods, with comparable effectiveness. Our new algorithms (LITE and LINE), demonstrate that simple stochastic selection can match complex optimizers like DEHB. This work explains why simple methods succeed in SE-real data occupies a small corner of possibilities-and guides when to apply them, challenging the need for CPU-heavy optimization. Our data and code are public at GitHub (see anon-artifacts/bingo).

Figures

Figures reproduced from arXiv: 2506.04509 by the authors.

Figure 1
Figure 1. Defect prediction: likelihood of a learner perform [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Number of configuration parameters over time (left). Usage of those parameters (right). From [30]. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. An example MOOT dataset (skipping middle rows). [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Algorithm 1, applied to Table I. Buckets (after) counts the filled buckets seen after increasing the data divisions. Y-axis shows mean results for data sets grouped by log2 of the number of rows. Blue lines indicate dataset sizes. Red lines show the bucket counts from …
Figure 5
Figure 5. Figure 5: For each sample size, in what percentage of the data [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Runtime comparison (log scale, milliseconds) of different optimizers across datasets. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When to Use Which? Benchmarking Optimisers for Configurable Systems under Varying Budgets

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Across 22 configurable systems and budgets from 100 to 10,000 evaluations, FLASH is the most consistently effective optimiser, while GA and IRACE catch up only at large budgets.

Reference graph

Works this paper leans on

80 extracted references · 74 canonical work pages · cited by 1 Pith paper

  1. [1]

    On the value of user preferences in search-based software engineering: A case study in software product lines,

    A. S. Sayyad, T. Menzies, and H. Ammar, “On the value of user preferences in search-based software engineering: A case study in software product lines,” in Proc. ICSE, 2013, pp. 492–501

    work page 2013

  2. [2]

    A case for automated configuration of variability-intensive systems,

    S. Apel, N. Siegmund, C. K ¨astner, and A. Legay, “A case for automated configuration of variability-intensive systems,” IEEE Software, vol. 37, no. 3, pp. 26–33, 2020

    work page 2020

  3. [3]

    Search-based software engineering in the era of modern software systems,

    F. Sarro, “Search-based software engineering in the era of modern software systems,” in Proc. RE, 2023, pp. 3–5

    work page 2023

  4. [4]

    The business case for automated software engineering,

    T. Menzies, O. Elrawas, J. Hihn, M. Feather, R. Madachy, and B. Boehm, “The business case for automated software engineering,” in Proc. ICSE, 2007, p. 303–312

    work page 2007

  5. [5]

    Optimizing energy consumption of guis in android apps: A multi-objective approach,

    M. Linares-V ´asquez, G. Bavota, C. E. B. C ´ardenas, R. Oliveto, M. Di Penta, and D. Poshyvanyk, “Optimizing energy consumption of guis in android apps: A multi-objective approach,” in Proc. ESEC/FSE, 2015, pp. 143–154

    work page 2015

  6. [6]

    Active learning for multi-objective optimization,

    M. Zuluaga, G. Sergent, A. Krause, and M. P ¨uschel, “Active learning for multi-objective optimization,” in International conference on machine learning. PMLR, 2013, pp. 462–470

    work page 2013

  7. [7]

    Dehb: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization,

    N. Awad, N. Mallik, and F. Hutter, “Dehb: Evolutionary hyberband for scalable, robust and efficient hyperparameter optimization,” in Proc. IJCAI, 8 2021, pp. 2147–2153

    work page 2021

  8. [8]

    Improving vulnera- bility inspection efficiency using active learning,

    Z. Yu, C. Theisen, L. Williams, and T. Menzies, “Improving vulnera- bility inspection efficiency using active learning,” IEEE Transactions on Software Engineering, vol. 47, no. 11, pp. 2401–2420, 2019

    work page 2019

Show all 80 references
  1. [9]

    Frugal: Unlocking ssl for software analytics,

    H. Tu and T. Menzies, “Frugal: Unlocking ssl for software analytics,” in ASE’21, 2021

    2021

  2. [10]

    When less is more: on the value of “co-training

    S. Majumder, J. Chakraborty, and T. Menzies, “When less is more: on the value of “co-training” for semi-supervised software defect predictors,” Empirical Software Engineering , vol. 29, no. 2, p. 51, 2024

    2024

  3. [11]

    Active learning and effort estimation: Finding the essential content of software effort estimation data,

    E. Kocaguneli, T. Menzies, J. Keung, D. Cok, and R. Madachy, “Active learning and effort estimation: Finding the essential content of software effort estimation data,” IEEE Transactions on Software Engineering , vol. 39, no. 8, pp. 1040–1053, 2012

    2012

  4. [12]

    Finding the right data for software cost modeling,

    Z. Chen, T. Menzies, D. Port, and D. Boehm, “Finding the right data for software cost modeling,” IEEE software, vol. 22, no. 6, pp. 38–46, 2005

    2005

  5. [13]

    Implications of ceiling effects in defect predictors,

    T. Menzies, B. Turhan, A. Bener, G. Gay, B. Cukic, and Y . Jiang, “Implications of ceiling effects in defect predictors,” in Proc. PROMISE. ACM, 2008, pp. 47–54

    2008

  6. [14]

    Semi-supervised learning literature survey,

    X. Zhu, “Semi-supervised learning literature survey,” Computer Sciences Technical Report, vol. 1530, pp. 1–59, 2005

    2005

  7. [15]

    Extensions of lipschitz mappings into a hilbert space,

    W. B. Johnson and J. Lindenstrauss, “Extensions of lipschitz mappings into a hilbert space,” Contemporary Mathematics, vol. 26, pp. 189–206, 1984

    1984

  8. [16]

    Benchmarking attribute selection techniques for discrete class data mining,

    M. A. Hall and G. Holmes, “Benchmarking attribute selection techniques for discrete class data mining,” IEEE Transactions on Knowledge and Data engineering, vol. 15, no. 6, pp. 1437–1447, 2003

    2003

  9. [17]

    How to” dodge

    A. Agrawal, W. Fu, D. Chen, X. Shen, and T. Menzies, “How to” dodge” complex software analytics,” IEEE Trans. Softw. Eng. , vol. 47, no. 10, pp. 2182–2194, 2019

    2019

  10. [18]

    Optimizing predictions for very small data sets: a case study on open-source project health prediction,

    A. Lustosa and T. Menzies, “Optimizing predictions for very small data sets: a case study on open-source project health prediction,” arXiv preprint arXiv:2301.06577, 2023

    2023 arXiv

  11. [19]

    Random search for hyper-parameter opti- mization

    J. Bergstra and Y . Bengio, “Random search for hyper-parameter opti- mization.” Journal of machine learning research , vol. 13, no. 2, 2012

    2012

  12. [20]

    Can large language models improve se active learning via warm-starts?

    L. Senthilkumar and T. Menzies, “Can large language models improve se active learning via warm-starts?” arXiv preprint arXiv:2501.00125 , 2024

    2024 arXiv

  13. [21]

    A survey on deep tabular learning,

    S. Somvanshi, S. Das, S. A. Javed, G. Antariksa, and A. Hossain, “A survey on deep tabular learning,” arXiv preprint arXiv:2410.12034 , 2024

    2024 arXiv

  14. [22]

    Automated parameter optimization of classification techniques for defect prediction models,

    C. Tantithamthavorn, S. McIntosh, A. E. Hassan, and K. Matsumoto, “Automated parameter optimization of classification techniques for defect prediction models,” in Proceedings of the 38th International Conference on Software Engineering , 2016, pp. 321–332

    2016

  15. [23]

    The ”bigse

    R. Krishna, Z. Yu, A. Agrawal, M. Dominguez, and D. Wolf, “The ”bigse” project: Lessons learned from validating industrial text mining,” in Proc. BIGDSE, 2016, p. 65–71

    2016

  16. [24]

    Tuning for software analytics: Is it really necessary?

    W. Fu, T. Menzies, and X. Shen, “Tuning for software analytics: Is it really necessary?” Inf. Softw. Technol., vol. 76, pp. 135–146, 2016

    2016

  17. [25]

    On the value of oversampling for deep learning in software defect prediction,

    R. Yedida and T. Menzies, “On the value of oversampling for deep learning in software defect prediction,” IEEE Trans. Soft. Eng. , vol. 48, no. 8, pp. 3103–3116, 2021

    2021

  18. [26]

    How to find actionable static analysis warnings: A case study with findbugs,

    R. Yedida, H. J. Kang, H. Tu, X. Yang, D. Lo, and T. Menzies, “How to find actionable static analysis warnings: A case study with findbugs,” IEEE Trans. Soft. Eng. , 2023

    2023

  19. [27]

    Performance- influence models for highly configurable systems,

    N. Siegmund, A. Grebhahn, S. Apel, and C. K ¨astner, “Performance- influence models for highly configurable systems,” in Proc. ESEC/FSE, 2015, pp. 284–294

    2015

  20. [28]

    Personal communication with author, at icse’25, ottawa, canda,

    J. Doe, “Personal communication with author, at icse’25, ottawa, canda,” Email to Your Name on May 15, 2023, May 2023, regarding the results of the K-means++ experiment

    2023

  21. [29]

    Finding faster configurations using flash,

    V . Nair, Z. Yu, T. Menzies, N. Siegmund, and S. Apel, “Finding faster configurations using flash,” IEEE Transactions on Software Engineering, vol. 46, no. 7, pp. 794–811, 2018

    2018

  22. [30]

    Hey, you have given me too many knobs!

    T. Xu, L. Jin, X. Fan, Y . Zhou, S. Pasupathy, and R. Talwadker, “Hey, you have given me too many knobs!” in Proc. FSE, 2015, pp. 307–319

    2015

  23. [31]

    Understanding and detecting software configuration errors,

    Y . Z. et al., “Understanding and detecting software configuration errors,” USENIX OSDI, 2011

    2011

  24. [32]

    Automatic database management system tuning through large-scale machine learning,

    D. V . Aken, A. Pavlo, G. J. Gordon, and B. Zhang, “Automatic database management system tuning through large-scale machine learning,” in SIGMOD, 2017

    2017

  25. [33]

    Starfish: A self-tuning system for big data analytics,

    H. Herodotou and S. Babu, “Starfish: A self-tuning system for big data analytics,” in Proceedings of the 5th Biennial Conference on Innovative Data Systems Research (CIDR ’11), Asilomar, CA, USA, January 9-12, 2011 , 2011, pp. 261–272. [Online]. Available: http://www.cidrdb.or...

    2011

  26. [34]

    Uncertainty-aware self-adaptation in cloud computing,

    P. Jamshidi and G. Casale, “Uncertainty-aware self-adaptation in cloud computing,” Ph.D. dissertation, Imperial College London, 2016, ph.D. dissertation proposal. Cited by other works for the Storm configuration performance example

    2016

  27. [35]

    How to evaluate solutions in pareto-based search-based software engineering: A critical review and methodological guidance,

    M. Li, T. Chen, and X. Yao, “How to evaluate solutions in pareto-based search-based software engineering: A critical review and methodological guidance,” IEEE Transactions on Software Engineering , vol. 48, no. 5, pp. 1771–1799, 2022

    2022

  28. [36]

    Just-in-time bug prediction in mobile applications: the domain matters!

    G. Catolino, “Just-in-time bug prediction in mobile applications: the domain matters!” in 2017 IEEE/ACM 4th International Conference on Mobile Software Engineering and Systems (MOBILESoft). IEEE, 2017, pp. 201–202

    2017

  29. [37]

    What do large commits tell us?: a taxonomical study of large commits,

    A. Hindle, D. M. German, and R. Holt, “What do large commits tell us?: a taxonomical study of large commits,” in Proceedings of the 2008 international working conference on Mining software repositories . ACM, 2008, pp. 99–108

    2008

  30. [38]

    A large-scale empirical study of just-in-time quality assurance,

    Y . Kamei, E. Shihab, B. Adams, A. E. Hassan, A. Mockus, A. Sinha, and N. Ubayashi, “A large-scale empirical study of just-in-time quality assurance,” IEEE Transactions on Software Engineering , vol. 39, no. 6, pp. 757–773, 2012

    2012

  31. [39]

    Classifying software changes: Clean or buggy?

    S. Kim, E. J. Whitehead Jr, and Y . Zhang, “Classifying software changes: Clean or buggy?” IEEE Transactions on Software Engineering , vol. 34, no. 2, pp. 181–196, 2008

    2008

  32. [40]

    Identifying reasons for software changes using historic databases

    A. Mockus and L. G. V otta, “Identifying reasons for software changes using historic databases.” in icsm, 2000, pp. 120–130

    2000

  33. [41]

    Personnel communication at fse’18,

    B. Vasilescu, “Personnel communication at fse’18,” Found. Softw. Eng, 2018

    2018

  34. [42]

    Quality and productivity outcomes relating to continuous integration in github,

    B. Vasilescu, Y . Yu, H. Wang, P. Devanbu, and V . Filkov, “Quality and productivity outcomes relating to continuous integration in github,” in Proceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering. ACM, 2015, pp. 805–816

    2015

  35. [43]

    Identifying self-admitted technical debts with jitterbug: A two-step approach,

    Z. Yu, F. M. Fahid, H. Tu, and T. Menzies, “Identifying self-admitted technical debts with jitterbug: A two-step approach,” IEEE Transactions on Software Engineering , vol. 48, no. 5, pp. 1676–1691, 2022

    2022

  36. [44]

    Data quality matters: A case study on data label correctness for security bug report prediction,

    X. Wu, W. Zheng, X. Xia, and D. Lo, “Data quality matters: A case study on data label correctness for security bug report prediction,” IEEE Transactions on Software Engineering , vol. 48, no. 7, pp. 2541–2556, 2022

    2022

  37. [45]

    Detecting false alarms from automatic static analysis tools: How far are we?

    H. J. Kang, K. L. Aw, and D. Lo, “Detecting false alarms from automatic static analysis tools: How far are we?” in Proceedings of the 44th International Conference on Software Engineering , ser. ICSE ’22. New York, NY , USA: Association for Computing Machinery, 2022, p. 698–70...

    2022

  38. [46]

    DBSCAN revisited, revisited: Why and how you should (still) use DBSCAN,

    E. Schubert, J. Sander, M. Ester, H.-P. Kriegel, and X. Xu, “DBSCAN revisited, revisited: Why and how you should (still) use DBSCAN,” ACM Transactions on Database Systems (TODS) , vol. 42, no. 3, pp. 19:1–19:21, 2017

    2017

  39. [47]

    Principal components analysis,

    K. Pearson, “Principal components analysis,” The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science , vol. 6, no. 2, p. 559, 1901

    1901

  40. [48]

    Faloutsos and K.-I

    C. Faloutsos and K.-I. Lin, FastMap: A fast algorithm for indexing, data-mining and visualization of traditional and multimedia datasets . ACM, 1995, vol. 24, no. 2

    1995

  41. [49]

    Instance-based learning algorithms,

    D. W. Aha, D. Kibler, and M. K. Albert, “Instance-based learning algorithms,” Machine learning, vol. 6, no. 1, pp. 37–66, 1991

    1991

  42. [50]

    Active learning,

    B. Settles, “Active learning,” Synthesis Lectures on Artificial Intelligence and Machine Learning , 2012

    2012

  43. [51]

    Hyper- band: A novel bandit-based approach to hyperparameter optimization,

    L. Li, K. Jamieson, G. DeSalvo, R. Ranganath, and R. Bardenet, “Hyper- band: A novel bandit-based approach to hyperparameter optimization,” The Journal of Machine Learning Research , vol. 18, no. 1, pp. 1–52, 2018

    2018

  44. [52]

    Non-stochastic best arm identification and hyperparameter optimization,

    K. Jamieson and A. Talwalkar, “Non-stochastic best arm identification and hyperparameter optimization,” in Proc. AISTATS, ser. Proceedings of Machine Learning Research, vol. 51, 2016, pp. 240–248

    2016

  45. [53]

    Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces,

    R. Storn and K. Price, “Differential evolution–a simple and efficient heuristic for global optimization over continuous spaces,” Journal of global optimization, vol. 11, no. 4, pp. 341–359, 1997

    1997

  46. [54]

    Hyperopt: a python library for model selection and hyperparameter optimization,

    J. Bergstra, B. Komer, C. Eliasmith, D. Yamins, and D. D. Cox, “Hyperopt: a python library for model selection and hyperparameter optimization,” Computational Science & Discovery , vol. 8, no. 1, p. 014008, 2015

    2015

  47. [55]

    Algorithms for hyper- parameter optimization,

    J. Bergstra, R. Bardenet, Y . Bengio, and B. K ´egl, “Algorithms for hyper- parameter optimization,” in Proc. NeurIPS, 2011, pp. 2546–2554

    2011

  48. [56]

    Predictive accuracy-based active learning for medical image segmentation,

    J. Shi, S. Ruan, Z. Zhu, M. Zhao, H. An, X. Xue, and B. Yan, “Predictive accuracy-based active learning for medical image segmentation,” in Proc. IJCAI, 2024, pp. 4885–4893

    2024

  49. [57]

    Cluster-based active learning,

    N. Yehuda and Y . Hacohen, “Cluster-based active learning,” in Bayesian Deep Learning Workshop at NeurIPS , 2018. [Online]. Available: http://bayesiandeeplearning.org/2018/papers/23.pdf

    2018

  50. [58]

    k-means++: The advantages of careful seeding,

    D. Arthur and S. Vassilvitskii, “k-means++: The advantages of careful seeding,” in Proc. SODA, ser. SODA ’07, 2007, p. 1027–1035

    2007

  51. [59]

    Active learning literature survey,

    B. Settles, “Active learning literature survey,” University of Wisconsin– Madison, Tech. Rep. 1648, 2009

    2009

  52. [60]

    C. E. Rasmussen and C. I. Williams, Gaussian Processes for Machine Learning. MIT Press, 2006

    2006

  53. [61]

    Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization,

    F. Pfisterer, L. Schneider, J. Moosbauer, M. Binder, and B. Bischl, “Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization,” in International Conference on Auto- mated Machine Learning . PMLR, 2022, pp. 3–1

    2022

  54. [62]

    Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks,

    A. Zela, J. N. Siems, L. Zimmer, J. Lukasik, M. Keuper, and F. Hutter, “Surrogate NAS benchmarks: Going beyond the limited search spaces of tabular NAS benchmarks,” in ICLR, 2022

    2022

  55. [63]

    A cluster analysis method for grouping means in the analysis of variance,

    A. J. Scott and M. Knott, “A cluster analysis method for grouping means in the analysis of variance,” Biometrics, vol. 30, pp. 507–512, 1974

    1974

  56. [64]

    Cliff’s delta calculator: A non-parametric effect size program for two groups of observations,

    G. Macbeth, E. Razumiejczyk, and R. D. Ledesma, “Cliff’s delta calculator: A non-parametric effect size program for two groups of observations,” Universitas Psychologica , vol. 10, no. 2, pp. 545–555, 2011

    2011

  57. [65]

    Efron and R

    B. Efron and R. J. Tibshirani, An Introduction to the Bootstrap , ser. Monographs on Statistics and Applied Probability. Boca Raton, Florida, USA: Chapman & Hall/CRC, 1993, no. 57

    1993

  58. [66]

    Distribution-free multiple comparisons,

    P. B. Nemenyi, “Distribution-free multiple comparisons,” Ph.D. disser- tation, Princeton University, 1963

    1963

  59. [67]

    Moea/d: A multiobjective evolutionary algorithm based on decomposition,

    Q. Zhang and H. Li, “Moea/d: A multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on evolutionary computa- tion, vol. 11, no. 6, pp. 712–731, 2007

    2007

  60. [68]

    Priorband: practical hyperparameter optimiza- tion in the age of deep learning,

    N. Mallik, E. Bergman, C. Hvarfner, D. Stoll, M. Janowski, M. Lindauer, L. Nardi, and F. Hutter, “Priorband: practical hyperparameter optimiza- tion in the age of deep learning,” in Proc. NeurIPS, ser. NIPS ’23, 2023

    2023

  61. [69]

    A survey on deep learning for software engineering,

    Y . Yang, X. Xia, D. Lo, and J. Grundy, “A survey on deep learning for software engineering,” ACM Computing Surveys (CSUR) , vol. 54, no. 10s, pp. 1–73, 2022

    2022

  62. [70]

    Veer: enhancing the interpretability of model-based optimizations,

    K. Peng, C. Kaltenecker, N. Siegmund, S. Apel, and T. Menzies, “Veer: enhancing the interpretability of model-based optimizations,” Empirical Software Engineering, vol. 28, no. 3, p. 61, 2023

    2023

  63. [71]

    “sampling

    J. Chen, V . Nair, R. Krishna, and T. Menzies, ““sampling” as a baseline optimizer for search-based software engineering,” IEEE Transactions on Software Engineering, vol. 45, no. 6, pp. 597–614, 2019

    2019

  64. [72]

    Sequential model optimization for software effort estimation,

    T. Xia, R. Shu, X. Shen, and T. Menzies, “Sequential model optimization for software effort estimation,” IEEE Transactions on Software Engi- neering, vol. 48, no. 6, pp. 1994–2009, 2020

    1994

  65. [73]

    Data-efficient performance learning for configurable systems,

    J. Guo, D. Yang, N. Siegmund, S. Apel, A. Sarkar, P. Valov, K. Czar- necki, A. Wasowski, and H. Yu, “Data-efficient performance learning for configurable systems,” Empirical Software Engineering, vol. 23, pp. 1826–1867, 2018

    2018

  66. [74]

    Learning from very little data: On the value of landscape analysis for predicting software project health,

    A. Lustosa and T. Menzies, “Learning from very little data: On the value of landscape analysis for predicting software project health,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 3, pp. 1–22, 2024

    2024

  67. [75]

    Trading off scalability, privacy, and performance in data synthesis,

    X. Ling, T. Menzies, C. Hazard, J. Shu, and J. Beel, “Trading off scalability, privacy, and performance in data synthesis,” IEEE Access , vol. 12, pp. 26 642–26 654, 2024

    2024

  68. [76]

    “the best data are fake data?

    T. Menzies and C. Hazard, ““the best data are fake data?”: An interview with chris hazard,” IEEE Software, vol. 40, no. 5, pp. 121–124, 2023

    2023

  69. [77]

    Smac3: A versatile bayesian optimization package for hyperparameter optimization,

    M. Lindauer, K. Eggensperger, M. Feurer, A. Biedenkapp, D. Deng, C. Benjamins, T. Ruhkopf, R. Sass, and F. Hutter, “Smac3: A versatile bayesian optimization package for hyperparameter optimization,” Jour- nal of Machine Learning Research , vol. 23, no. 54, pp. 1–9, 2022

    2022

  70. [78]

    Silberschatz, P

    A. Silberschatz, P. B. Galvin, and G. Gagne, Operating System Concepts. Wiley, 2018

    2018

  71. [79]

    The psycho-biology of language, gk zipf, mit press, cam- bridge, ma.(1935),

    A. Mellor, “The psycho-biology of language, gk zipf, mit press, cam- bridge, ma.(1935),” 2006

    1935

  72. [80]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems , 2017, pp. 5998–6008

    2017

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.