Pith. sign in

REVIEW 3 major objections 5 minor 83 references

VirnyFlow: A Design Space for Responsible Model Development

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VirnyFlow is a design space that lets developers optimize fairness and stability together with accuracy, and outperforms accuracy-only AutoML on combined quality and scale.

desk verdict Real systems paper, but the headline quality comparison against AutoML baselines is not supported by the experiments as run. read the letter →

arxiv 2506.01584 v1 pith:4XDH2CGN submitted 2025-06-02 cs.LG cs.AIcs.CY

Denys Herasymuk , Nazar Protsiv , Julia Stoyanovich This is my paper
classification cs.LGcs.AIcs.CY
keywords responsiblemodeldevelopmentAutoMLmulti-objectiveBayesianoptimizationfairness-awarestabilitypipelinecost-awarebanditsdistributedexecution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper presents VirnyFlow, a system designed as a design space for responsible model development rather than a black-box AutoML tool. Its central claim is that letting a data scientist specify an evaluation protocol—which metrics matter, on which demographic groups (including intersections), and with which weights—and then searching over the whole pipeline (null imputation, fairness interventions, model choice) with multi-objective Bayesian optimization produces better pipelines than optimizing accuracy alone. On five tabular benchmarks drawn from hiring, healthcare, and public coverage, VirnyFlow's best pipelines achieve a higher equal-weighted average of accuracy, fairness, and stability than Alpine Meadow and auto-sklearn, and scale to 128 workers while those systems are capped at one node. If correct, this makes responsible ML development a tunable, interactive design problem instead of a single-metric automation problem.

What carries the argument

The load-bearing machinery is the integration of a user-defined evaluation protocol with a two-level optimizer. At the logical level, a multi-armed bandit scores each logical pipeline—a DAG of pipeline primitives with hyperparameters left open—using $s=\sum_i w_i\,\mu_i + \frac{\theta}{c}\sum_i w_i\,\delta_i$, where $w_i$, $\mu_i$, and $\delta_i$ are the weight, mean, and standard deviation of the $i$-th objective and $c$ is the pipeline's observed cost; the risk factor $\theta$ controls tolerance for variance and the exploration factor $\beta$ controls random exploration. At the physical level, multi-objective Bayesian optimization with the EHVI acquisition function generates complete pipelines with fixed hyperparameters, tuning multiple stages simultaneously. An extended Adaptive Pipeline Selection algorithm prunes unpromising pipelines on partial training data using a weighted multi-objective error, and a Kafka- and MongoDB-based distributed architecture lets the search continue asynchronously across many workers.

What would settle it

Re-run the comparison on the three main benchmark datasets with the same 60-minute budget and resources, but tell Alpine Meadow and auto-sklearn to optimize the same weighted multi-objective combination VirnyFlow uses (equal weights on the F1 score, the dataset's fairness metric, and label stability); if either baseline matches or exceeds VirnyFlow's equal-weighted average score, the paper's headline claim would be refuted. A cheaper check is to recompute Table 2's averages while holding out the fairness and stability metrics that the baselines never optimized and see whether the advantage persists.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that responsibility-related objectives can be put into the optimization loop itself, across the full ML lifecycle, without giving up competitive predictive performance. VirnyFlow treats an experiment as a configurable design space: an evaluation protocol (metrics, groups, weights, reference point) drives a two-level search in which a cost-aware multi-armed bandit selects promising logical pipelines and a multi-objective Bayesian optimizer instantiates physical pipelines that jointly tune imputation, fairness interventions, and model hyperparameters. The empirical claim, supported by Table 2 and Figures 4–6, is that this system outperforms Alpine Meadow and auto-sklearn on the equal-weighted average of the chosen metrics on diabetes, folk-emp, and folk-pubcov, and reaches speedups of up to 7.07x and 5.03x on two larger benchmarks.

Load-bearing premise

The central performance claim rests on judging VirnyFlow and its baselines by the same equally weighted average of accuracy, fairness, and stability, even though the baselines were instructed to optimize accuracy alone; if that comparison is not accepted as fair, the outperformance claim collapses.

Editorial extensions

If this is right

  • Users can treat fairness and stability as first-class, weighted objectives alongside accuracy, including for intersectional groups such as sex combined with race, rather than hoping they improve incidentally.
  • Pipeline stages that are usually fixed in AutoML—null imputation, fairness interventions, model family—are tuned jointly, so trade-offs among them are visible in the optimization results.
  • The comparison protocol of equal budget and search space, judged by an equal-weight average over F1, fairness, and stability, gives a reproducible way to benchmark future responsible-AutoML systems.
  • The scalability results imply that distributed execution can extend multi-objective pipeline search to larger tabular datasets, with measured speedups up to 7.07x on a 70K-row dataset and 5.03x on a 200K-row dataset at 128 workers.
  • Sensitivity experiments suggest practical defaults—around 4 physical pipeline candidates per selection and {0.5, 1.0} or {0.75, 1.0} pruning fractions—as a starting point for new experiments.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline comparison rewards VirnyFlow for optimizing fairness and stability while its baselines were told to optimize accuracy only; a stronger test would give the baselines the same weighted multi-objective objective and see whether the average-score gap persists.
  • The same logical/physical pipeline separation could carry other lifecycle concerns—robustness to distribution shift, calibration, privacy loss, or energy cost—by adding them as objectives in the same evaluation protocol; the paper does not claim this.
  • Because the baselines are limited to single-node execution, the scalability gap is partly architectural; a distributed re-implementation of a baseline could narrow it, and this comparison is not made in the paper.
  • The interactive visualization layer suggests a human-in-the-loop evaluation—measuring whether users choose better metric weights or intervene more effectively with VirnyFlow than without it—which the paper motivates but does not test.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents VirnyFlow, a system/design space for responsible ML pipeline development that lets users define context-specific multi-objective criteria (accuracy, fairness, stability), search over multi-stage pipelines (imputation, fairness interventions, models), and execute experiments in a distributed manner using multi-objective Bayesian optimization, cost-aware bandits, query optimization, and Apache Kafka/MongoDB-based parallelism. The evaluation has three parts: (RQ1) case studies showing that VirnyFlow can optimize different weightings of F1, fairness metrics (TPRD, FNRD, SRD), and Label Stability on three tabular datasets; (RQ2) a comparison with Alpine Meadow and auto-sklearn on optimization quality (three datasets, Table 2) and scalability (two datasets, Figure 4); and (RQ3) a sensitivity analysis of the number of pipeline candidates and pruning fractions. The abstract claims that VirnyFlow significantly outperforms state-of-the-art AutoML systems in both optimization quality and scalability across five real-world benchmarks.

Significance. If the comparative claims were supported, this would be a valuable contribution: VirnyFlow integrates intersectional fairness and stability into multi-objective AutoML, provides a flexible experiment-protocol layer, and supports distributed execution. The RQ1 case studies (Section 3.2, Figure 3) are informative and show plausible trade-offs between accuracy, fairness, and stability, and the authors provide a public repository with configurations. However, the headline comparative claims are not currently supported because the evaluation protocol in Section 3.3 gives VirnyFlow a multi-objective objective while baselines optimize F1 alone, and the speedup metric in Sections 3.1 and 3.4 is nonstandard and may mislead. The system-design contribution is real, but the empirical validation needs substantial reworking before the central claims can be accepted.

major comments (3)
  1. [Section 3.3, Table 2]
  2. [Sections 3.1 and 3.4, Figure 4]
  3. [Section 2.5, Algorithm 1]
minor comments (5)
  1. [Abstract and Section 3.4]
  2. [Table 2]
  3. [Section 3.5, Table 3]
  4. [Section 3.2]
  5. [Section 3.4]

Circularity Check

2 steps flagged · score 6.0 of 10

The headline quality comparison is an artifact of the evaluation protocol: VirnyFlow optimizes the equal-weighted multi-objective score that is then used as the yardstick, while the baselines are restricted to F1 alone.

  1. self definitional [Section 3.3 (Performance of VirnyFlow vs. Other Systems), Table 2; Section 3.1 (Metrics for Comparison)]
    "Here, VirnyFlow performs multi-objective optimization using equal weights for each selected metric, while Alpine Meadow and auto-sklearn optimize solely for F1. ... Bold text indicates the best value for individual metrics, while gray shading highlights the best average score across multiple metrics using equal weighting. The results show that VirnyFlow consistently outperforms both Alpine Meadow and auto-sklearn in terms of the average score across all three datasets."

    The yardstick used to declare VirnyFlow superior—the 'average score across multiple metrics using equal weighting'—is exactly the scalarized objective that VirnyFlow was allowed to optimize (equal weights on F1 plus fairness/stability), while the baselines were restricted to F1 alone. The claimed quality advantage is therefore not an independent measurement of optimization ability: it re-scores all systems on VirnyFlow's own objective. The folk-pubcov row makes the mechanism explicit: VirnyFlow's F1 (0.596) is below auto-sklearn (0.633) and Alpine Meadow (0.628), yet it wins because the fairness terms it alone optimized dominate the average.

  2. other [Section 3.1 (Metrics for Comparison) and Figure 4 (Scalability study)]
    "Speedup is defined as the ratio between the runtime of VirnyFlow using a single worker/CPU and the runtime of the system using k workers/CPUs. For consistency, when computing speedup for Alpine Meadow and auto-sklearn, we also use VirnyFlow's single-worker runtime as the baseline to ensure a uniform basis for comparison."

    The quantity reported as 'speedup' for Alpine Meadow and auto-sklearn is not their own speedup; it is VirnyFlow's single-worker runtime divided by the baseline's k-worker runtime. Thus the scalability comparison in Figure 4 is normalized by VirnyFlow's serial performance, so the baseline curves are not independent measurements of the baselines' scaling behavior. The reported superiority in speedup is therefore partly a function of VirnyFlow's own single-worker runtime rather than a neutral comparison of each system's parallel scaling.

full rationale

VirnyFlow's architecture—multi-stage multi-objective Bayesian optimization, cost-aware bandits, adaptive pipeline pruning, and distributed Kafka/MongoDB execution—is a real engineering contribution and is not derived from the benchmarks; those design claims are self-contained. The circularity is confined to the empirical validation of the headline performance claims. The quality comparison uses an evaluation score identical to VirnyFlow's own scalarized objective while denying the baselines that objective, so the 'significantly outperforms' result is an artifact of the protocol rather than evidence of superior optimization quality. The speedup metric is also defined relative to VirnyFlow's serial runtime for all systems, making the scalability comparison self-referential. Self-citations to Virny and the Label Stability paper are present, but the stability metric and evaluation library are external definitions and do not by themselves make the derivation circular. Overall, the central empirical claim reduces by construction to the chosen comparison protocol, while the system's functionality retains independent content.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several assumptions about evaluation metrics, the validity of the comparison protocol, and the correctness of extending the pruning algorithm. None of these are derived in the paper; they are asserted or inherited from prior work. The free parameters are user configuration choices, not fitted values. No new entities are postulated.

free parameters (6)
  • exploration_factor = 0.5 (Listing 1)
    Controls probability of selecting top-scoring versus random logical pipeline; user-defined, not fitted to data.
  • risk_factor = 0.5
    Scales variance term in logical pipeline score Eq. 1; user-defined.
  • number_of_pipeline_candidates_k = 2 in Section 3.2; 4 recommended in Table 3
    Number of physical pipeline suggestions per logical pipeline; affects parallelism and performance trade-off.
  • bootstrap_fraction = 0.8
    Bootstrap fraction for Label Stability estimation in Virny; user-set.
  • n_estimators = 50
    Number of bootstrap models for Label Stability; user-set.
  • training_fractions_for_halting = {0.7, 1.0} in Section 3.2; varied in Table 9
    Fractions of training data used for early pruning in APS; user-set.
assumptions (4)
  • domain assumption Label Stability, as computed from bootstrap disagreement, is a valid and optimizable measure of model stability.
    Introduced in prior work by the same group (refs [20,44]); the paper assumes it can serve as an optimization objective.
  • ad hoc to paper Equal-weighted average of F1, fairness, and stability metrics is the appropriate scalar to rank systems.
    Used in Table 2 to compare systems that optimized different objectives; the choice of weighting is not justified.
  • ad hoc to paper The multi-objective extension of APS, using a weighted sum of errors, preserves the validity of early pruning.
    Algorithm 1 extends Alpine Meadow's single-objective APS without a proof of correctness for the multi-objective case.
  • ad hoc to paper Baseline AutoML systems should optimize only F1 because they are not multi-objective.
    This choice makes the comparison favor the multi-objective system; not an external standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VirnyFlow: A Design Space for Responsible Model Development." pith.science (2026). https://pith.science/paper/4XDH2CGN

@misc{pith2026250601584,
  author       = {Pith},
  title        = {Pith review of: VirnyFlow: A Design Space for Responsible Model Development},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4XDH2CGN}},
  note         = {Machine review of arXiv:2506.01584}
}
read the original abstract

Developing machine learning (ML) models requires a deep understanding of real-world problems, which are inherently multi-objective. In this paper, we present VirnyFlow, the first design space for responsible model development, designed to assist data scientists in building ML pipelines that are tailored to the specific context of their problem. Unlike conventional AutoML frameworks, VirnyFlow enables users to define customized optimization criteria, perform comprehensive experimentation across pipeline stages, and iteratively refine models in alignment with real-world constraints. Our system integrates evaluation protocol definition, multi-objective Bayesian optimization, cost-aware multi-armed bandits, query optimization, and distributed parallelism into a unified architecture. We show that VirnyFlow significantly outperforms state-of-the-art AutoML systems in both optimization quality and scalability across five real-world benchmarks, offering a flexible, efficient, and responsible alternative to black-box automation in ML development.

Figures

Figures reproduced from arXiv: 2506.01584 by the authors.

Figure 2
Figure 2. The VirnyFlow architecture. MongoDB com￾ponents shown in green, Kafka queues in gray, work￾ers in yellow, and Task Manager modules in blue. Task Provider, Task Generator, and Cost Model Updater use OpenBox for multi-stage, multi-objective optimization. (1) Search space construction: The execution begins when a user provides an experiment config to Task Manager, similar to one in Listing 1, and defines a hyper-parame… view at source ↗
Figure 3
Figure 3. Mean and standard deviation of performance [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Scalability study with varying numbers of workers [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 5
Figure 5. Figure 5: Scalability study on heart (70K). A dashed line high￾lights the best value for TNRD. 0 10 20 30 40 50 60 70 80 90 100110120130 # of Workers 0 10 20 30 40 50 60 Speedup 5.03x virny_flow alpine_meadow autosklearn (a) Speedup # of Workers 0.810 0.812 0.814 0.816 0.818 0.8…
Figure 7
Figure 7. Figure 7: Optimization progress for Objective 1 (F1) for [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Optimization progress for Objective 2 (FNRD) for [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Parallel coordinates plot for folk-emp. A visualization shows used hyperparameters for the logical pipeline in each iteration and the respective outcome values of Objectives 1 and 2. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_9.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

83 extracted references · 52 canonical work pages

  1. [1]

    Meta AI. 2025. Ax: Adaptive Experimentation Platform. https://ax.dev. Accessed: 2025-03-01

    2025

  2. [2]

    2024.Apache Kafka: A Distributed Event Streaming Platform

    Apache Software Foundation. 2024.Apache Kafka: A Distributed Event Streaming Platform. https://kafka.apache.org/ Version 3.6

    2024

  3. [3]

    2019.Bayesian optimization and data science

    Francesco Archetti and Antonio Candelieri. 2019.Bayesian optimization and data science. Vol. 849. Springer

    2019

  4. [4]

    Falaah Arif Khan, Denys Herasymuk, Nazar Protsiv, and Julia Stoyanovich. 2025. Still More Shades of Null: An Evaluation Suite for Responsible Missing Value Imputation.Proc. VLDB Endow.(2025). https://arxiv.org/pdf/2409.07510 accepted for publication, to appear

    arXiv 2025

  5. [5]

    Falaah Arif Khan, Denys Herasymuk, and Julia Stoyanovich. 2023. On Fairness and Stability: Is Estimator Variance a Friend or a Foe?CoRRabs/2302.04525 (2023). https://doi.org/10.48550/ARXIV.2302.04525 arXiv:2302.04525

    work page Pith review arXiv doi:10.48550/arxiv.2302.04525 2023

  6. [6]

    Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. 2020. BoTorch: A framework for efficient Monte-Carlo Bayesian optimization.Advances in neural information processing systems33 (2020), 21524–21538

    2020

  7. [7]

    Mitra Baratchi, Can Wang, Steffen Limmer, Jan N van Rijn, Holger Hoos, Thomas Bäck, and Markus Olhofer. 2024. Automated machine learning: past, present and future.Artificial Intelligence Review57, 5 (2024), 1–88

    work page 2024

  8. [8]

    Rafael Barbudo, Sebastián Ventura, and José Raúl Romero. 2023. Eight years of AutoML: categorisation, review and trends.Knowledge and Information Systems 65, 12 (2023), 5097–5149

    work page 2023

Show all 83 references
  1. [9]

    2023.Fairness and machine learning: Limitations and opportunities

    Solon Barocas, Moritz Hardt, and Arvind Narayanan. 2023.Fairness and machine learning: Limitations and opportunities. MIT press

    2023

  2. [10]

    Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. 2019. Max-value entropy search for multi-objective Bayesian optimization.Advances in neural information processing systems32 (2019)

    2019

  3. [11]

    Syrine Belakaria, Aryan Deshwal, and Janardhan Rao Doppa. 2019. Max- value Entropy Search for Multi-Objective Bayesian Optimization. InAd- vances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.),...

    2019

  4. [12]

    Syrine Belakaria, Aryan Deshwal, Nitthilan Kannappan Jayakodi, and Janard- han Rao Doppa. 2020. Uncertainty-Aware Search Framework for Multi-Objective Bayesian Optimization.Proceedings of the AAAI Conference on Artificial Intelli- gence34, 06 (April 2020), 10044–10052. https:/...

    2020 doi

  5. [13]

    Emily Black, Klas Leino, and Matt Fredrikson. 2021. Selective ensembles for consistent predictions.arXiv preprint arXiv:2111.08230(2021)

    2021 arXiv

  6. [14]

    Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings.Advances in neural information processing systems 29 (2016)

    2016

  7. [15]

    2016.MCDA and Multiobjective Evolutionary Algorithms

    Juergen Branke. 2016.MCDA and Multiobjective Evolutionary Algorithms. Springer New York, New York, NY, 977–1008. https://doi.org/10.1007/978-1- 4939-3094-4_23

    2016 doi

  8. [16]

    Lars Buitinck, Gilles Louppe, Mathieu Blondel, Fabian Pedregosa, Andreas Mueller, Olivier Grisel, Vlad Niculae, Peter Prettenhofer, Alexandre Gramfort, Jaques Grobler, et al. 2013. API design for machine learning software: experiences from the scikit-learn project.arXiv prepri...

    2013 arXiv

  9. [17]

    Joy Buolamwini and Timnit Gebru. 2018. Gender shades: Intersectional accu- racy disparities in commercial gender classification. InConference on fairness, accountability and transparency. PMLR, 77–91

    2018

  10. [18]

    Toon Calders and Sicco Verwer. 2010. Three naive bayes approaches for discrimination-free classification.Data mining and knowledge discovery21 (2010), 277–292

    2010

  11. [19]

    Antonio Candelieri, Andrea Ponti, and Francesco Archetti. 2024. Fair and green hyperparameter optimization via multi-objective and multiple information source Bayesian optimization.Machine Learning113, 5 (2024), 2701–2731

    2024

  12. [20]

    Darling and David J

    Michael C. Darling and David J. Stracuzzi. 2018. Toward Uncertainty Quantifica- tion for Supervised Classification

    2018

  13. [21]

    Richeek Das and Samuel Dooley. 2023. Fairer and more accurate tabular models through nas.arXiv preprint arXiv:2310.12145(2023)

    2023 arXiv

  14. [22]

    Jeffrey Dastin. 2022. Amazon scraps secret AI recruiting tool that showed bias against women. InEthics of data and analytics. Auerbach Publications, 296–299

    2022

  15. [23]

    Frances Ding, Moritz Hardt, John Miller, and Ludwig Schmidt. 2021. Retiring adult: New datasets for fair machine learning.NeurIPS34 (2021), 6478–6490

    2021

  16. [24]

    Samuel Dooley, Rhea Sukthanker, John Dickerson, Colin White, Frank Hutter, and Micah Goldblum. 2024. Rethinking bias mitigation: Fairer architectures make for fairer face recognition.Advances in Neural Information Processing Systems36 (2024)

    2024

  17. [25]

    Michael TM Emmerich, Kyriakos C Giannakoglou, and Boris Naujoks. 2006. Single-and multiobjective evolutionary optimization assisted by Gaussian ran- dom field metamodels.IEEE Transactions on Evolutionary Computation10, 4 (2006), 421–439

    2006

  18. [26]

    Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. 2020. Autogluon-tabular: Robust and accurate automl for structured data.arXiv preprint arXiv:2003.06505(2020)

    2020 arXiv

  19. [27]

    A Feder Cooper, Solon Barocas, Christopher De Sa, and Siddhartha Sen. 2023. Variance, Self-Consistency, and Arbitrariness in Fair Classification.arXiv e-prints (2023), arXiv–2301

    2023

  20. [28]

    Michael Feldman, Sorelle A Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and removing disparate impact. Inproceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 259–268

    2015

  21. [29]

    Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter. 2022. Auto-sklearn 2.0: Hands-free automl via meta-learning. Journal of Machine Learning Research23, 261 (2022), 1–61

    2022

  22. [30]

    Peter I Frazier. 2018. Bayesian optimization. InRecent advances in optimization and modeling of contemporary problems. Informs, 255–278

    2018

  23. [31]

    Yarin Gal et al. 2016. Uncertainty in deep learning. (2016)

    2016

  24. [32]

    2023.Bayesian optimization

    Roman Garnett. 2023.Bayesian optimization. Cambridge University Press

    2023

  25. [33]

    Pieter Gijsbers, Marcos LP Bueno, Stefan Coors, Erin LeDell, Sébastien Poirier, Janek Thomas, Bernd Bischl, and Joaquin Vanschoren. 2024. Amlb: an automl benchmark.Journal of Machine Learning Research25, 101 (2024), 1–65

    2024

  26. [34]

    Shubha Guha, Falaah Arif Khan, Julia Stoyanovich, and Sebastian Schelter. 2024. Automated data cleaning can hurt fairness in machine learning-based decision making.IEEE Transactions on Knowledge and Data Engineering36, 12 (2024), 7368–7379

    2024

  27. [35]

    Isabelle Guyon, Lisheng Sun-Hosoya, Marc Boullé, Hugo Jair Escalante, Sergio Escalera, Zhengying Liu, Damir Jajetic, Bisakha Ray, Mehreen Saeed, Michèle Sebag, et al. 2019. Analysis of the automl challenge series.Automated Machine Learning177 (2019), 177–219

    2019

  28. [36]

    Moritz Hardt, Eric Price, and Nati Srebro. 2016. Equality of opportunity in supervised learning.Advances in neural information processing systems29 (2016)

    2016

  29. [37]

    Denys Herasymuk, Falaah Arif Khan, and Julia Stoyanovich. 2024. Responsible Model Selection with Virny and VirnyView. InCompanion of the 2024 Interna- tional Conference on Management of Data. 488–491

    2024

  30. [38]

    Kevin Jamieson and Ameet Talwalkar. 2016. Non-stochastic best arm identifi- cation and hyperparameter optimization. InArtificial intelligence and statistics. PMLR, 240–248

    2016

  31. [39]

    Jones, Matthias Schonlau, and William J

    Donald R. Jones, Matthias Schonlau, and William J. Welch. 1998.Journal of Global Optimization13, 4 (1998), 455–492. https://doi.org/10.1023/a:1008306431147

    1998 doi

  32. [40]

    Manu Joseph and Harsh Raj. 2022. GANDALF: gated adaptive network for deep automated learning of features.arXiv preprint arXiv:2207.08548(2022)

    2022 arXiv

  33. [41]

    Faisal Kamiran and Toon Calders. 2009. Classifying without discriminating. In 2009 2nd international conference on computer, control and communication. IEEE, 1–6

    2009

  34. [42]

    Toshihiro Kamishima, Shotaro Akaho, and Jun Sakuma. 2011. Fairness-aware learning through regularization approach. In2011 IEEE 11th international confer- ence on data mining workshops. IEEE, 643–650

    2011

  35. [43]

    Florian Karl, Tobias Pielok, Julia Moosbauer, Florian Pfisterer, Stefan Coors, Martin Binder, Lennart Schneider, Janek Thomas, Jakob Richter, Michel Lang, et al

  36. [44]

    Falaah Arif Khan, Denys Herasymuk, and Julia Stoyanovich. 2023. On Fair- ness and Stability: Is Estimator Variance a Friend or a Foe?arXiv preprint arXiv:2302.04525(2023)

    2023 arXiv

  37. [45]

    CR Komala, Ashok Kumar, N Hema, S Nagarani, Ajay Singh Yadav, M Rajendiran, R Srinivasan, and V Vijayan. 2024. Fair-AutoML: Enhancing fairness in machine learning predictions through automated machine learning and bias mitigation techniques. InAIP Conference Proceedings, Vol. ...

    2024

  38. [46]

    Yang Li, Yu Shen, Wentao Zhang, Yuanwei Chen, Huaijun Jiang, Mingchao Liu, Jiawei Jiang, Jinyang Gao, Wentao Wu, Zhi Yang, et al . 2021. Openbox: A generalized black-box optimization service. InProceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining....

    2021

  39. [47]

    Yan Li, Matthew Sperrin, Darren M Ashcroft, and Tjeerd Pieter Van Staa. 2020. Consistency of variety of machine learning and statistical models in predicting clinical risks of individual patients: longitudinal cohort study using cardiovascu- lar disease as exemplar.BMJ371 (2020)

    2020

  40. [48]

    Jonas Močkus. 1975. On Bayesian methods for seeking the extremum. InOp- timization Techniques IFIP Technical Conference Novosibirsk, July 1–7, 1974 6. Springer, 400–404

    1975

  41. [49]

    2024.MongoDB: The Developer Data Platform

    MongoDB, Inc. 2024.MongoDB: The Developer Data Platform. https://www. mongodb.com/ Version 7.0

    2024

  42. [50]

    Arvind Narayanan. 2018. 21 Fairness Definitions and Their Politics. Available at https://www.cs.princeton.edu/~arvindn/talks

    2018

  43. [51]

    Felix Neutatz, Marius Lindauer, and Ziawasch Abedjan. 2024. AutoML in heavily constrained applications.The VLDB Journal33, 4 (2024), 957–979

    2024

  44. [52]

    Giang Nguyen, Sumon Biswas, and Hridesh Rajan. 2023. Fix fairness, don’t ruin accuracy: Performance aware fairness repair using automl. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the 9 Foundations of Software Engineering. 502–514

    2023

  45. [53]

    Nikolay O Nikitin, Pavel Vychuzhanin, Mikhail Sarafanov, Iana S Polonskaia, Ilia Revin, Irina V Barabanova, Gleb Maximov, Anna V Kalyuzhnaya, and Alexan- der Boukhanovsky. 2022. Automated evolutionary approach for the design of composite machine learning pipelines.Future Gener...

    2022

  46. [54]

    Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. 2019. Dissecting racial bias in an algorithm used to manage the health of populations. Science366, 6464 (2019), 447–453

    2019

  47. [55]

    Randal S Olson, Nathan Bartley, Ryan J Urbanowicz, and Jason H Moore. 2016. Evaluation of a tree-based pipeline optimization tool for automating data science. InProceedings of the genetic and evolutionary computation conference 2016. 485– 492

    2016

  48. [56]

    Biswajit Paria, Kirthevasan Kandasamy, and Barnabás Póczos. 2020. A Flexible Framework for Multi-Objective Bayesian Optimization using Random Scalariza- tions. InProceedings of The 35th Uncertainty in Artificial Intelligence Conference (Proceedings of Machine Learning Research...

    2020

  49. [57]

    Valerio Perrone, Michele Donini, Muhammad Bilal Zafar, Robin Schmucker, Krishnaram Kenthapadi, and Cédric Archambeau. 2021. Fair bayesian optimiza- tion. InProceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society. 854–863

    2021

  50. [58]

    Florian Pfisterer, Stefan Coors, Janek Thomas, and Bernd Bischl. 2019. Multi- objective automatic machine learning with autoxgboostmc.arXiv preprint arXiv:1908.10796(2019)

    2019 arXiv

  51. [59]

    Florian Pfisterer, Lennart Schneider, Julia Moosbauer, Martin Binder, and Bernd Bischl. 2022. Yahpo gym-an efficient multi-objective multi-fidelity benchmark for hyperparameter optimization. InInternational Conference on Automated Machine Learning. PMLR, 3–1

    2022

  52. [60]

    Herilalaina Rakotoarison, Marc Schoenauer, and Michèle Sebag. 2019. Automated machine learning with monte-carlo tree search.arXiv preprint arXiv:1906.00170 (2019)

    2019 arXiv

  53. [61]

    Jake Robertson, Thorsten Schmidt, Frank Hutter, and Noor Awad. 2024. A Human-in-the-Loop Fairness-Aware Model Selection Framework for Complex Fairness Objective Landscapes. InProceedings of the AAAI/ACM Conference on AI, Ethics, and Society, Vol. 7. 1231–1242

    2024

  54. [62]

    Sebastian Schelter and Julia Stoyanovich. 2020. Taming Technical Bias in Machine Learning Pipelines.IEEE Data Eng. Bull.43, 4 (2020). http://sites.computer.org/ debull/A20dec/p39.pdf

    2020

  55. [63]

    Adams, and Nando de Fre- itas

    Bobak Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Fre- itas. 2016. Taking the Human Out of the Loop: A Review of Bayesian Optimization. Proc. IEEE104, 1 (2016), 148–175. https://doi.org/10.1109/JPROC.2015.2494218

    2016

  56. [64]

    Zeyuan Shang, Emanuel Zgraggen, Benedetto Buratti, Ferdinand Kossmann, Philipp Eichmann, Yeounoh Chung, Carsten Binnig, Eli Upfal, and Tim Kraska

  57. [65]

    Julia Stoyanovich, Serge Abiteboul, Bill Howe, H. V. Jagadish, and Sebastian Schelter. 2022. Responsible data management.Commun. ACM65, 6 (2022), 64–74. https://doi.org/10.1145/3488717

    2022 doi

  58. [66]

    Harini Suresh and John Guttag. 2021. A framework for understanding sources of harm throughout the machine learning life cycle. InProceedings of the 1st ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization. 1–9

    2021

  59. [67]

    Shinya Suzuki, Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, and Masayuki Karasuyama. 2020. Multi-objective Bayesian Optimization using Pareto-frontier Entropy. InProceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research), ...

    2020

  60. [68]

    Thomas Swearingen, Will Drevo, Bennett Cyphers, Alfredo Cuesta-Infante, Arun Ross, and Kalyan Veeramachaneni. 2017. ATM: A distributed, collaborative, scalable system for automated machine learning. In2017 IEEE international conference on big data (big data). IEEE, 151–162

    2017

  61. [69]

    Latanya Sweeney. 2013. Discrimination in online ad delivery.Commun. ACM56, 5 (2013), 44–54

    2013

  62. [70]

    Anique Tahir, Lu Cheng, and Huan Liu. 2023. Fairness through aleatoric uncer- tainty. InProceedings of the 32nd ACM International Conference on Information and Knowledge Management. 2372–2381

    2023

  63. [71]

    Rachael Tatman. 2017. Gender and dialect bias in YouTube’s automatic captions. InProceedings of the first ACL workshop on ethics in natural language processing. 53–59

    2017

  64. [72]

    Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. 2013. Auto-WEKA: Combined selection and hyperparameter optimization of classifica- tion algorithms. InProceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining. 847–855

    2013

  65. [73]

    Neha Prerna Tigga and Shruti Garg. 2020. Prediction of Type 2 Diabetes using Machine Learning Classification Methods.Procedia Computer Science167 (2020), 706–716. https://doi.org/10.1016/j.procs.2020.03.336 International Conference on Computational Intelligence and Data Science

    2020 doi

  66. [74]

    Handing Wang, Markus Olhofer, and Yaochu Jin. 2017. A mini-review on pref- erence modeling and articulation in multi-objective optimization: Current sta- tus and challenges.Complex &; Intelligent Systems3, 4 (Aug 2017), 233–245. https://doi.org/10.1007/s40747-017-0053-9

    2017 doi

  67. [75]

    Hilde Weerts, Florian Pfisterer, Matthias Feurer, Katharina Eggensperger, Edward Bergman, Noor Awad, Joaquin Vanschoren, Mykola Pechenizkiy, Bernd Bischl, and Frank Hutter. 2024. Can fairness be automated? Guidelines and opportunities for fairness-aware AutoML.Journal of Artif...

    2024

  68. [76]

    Steven Euijong Whang, Yuji Roh, Hwanjun Song, and Jae-Gil Lee. 2023. Data collection and quality challenges in deep learning: A data-centric ai perspective. The VLDB Journal32, 4 (2023), 791–813

    2023

  69. [77]

    Qingyun Wu and Chi Wang. 2021. FairAutoML: Embracing unfairness mitigation in AutoML.arXiv preprint arXiv:2111.06495(2021)

    2021 arXiv

  70. [78]

    Jie Xu, Yunyu Xiao, Wendy Hui Wang, Yue Ning, Elizabeth A Shenkman, Jiang Bian, and Fei Wang. 2022. Algorithmic fairness in computational medicine. EBioMedicine84 (2022)

    2022

  71. [79]

    Anatoly Yakovlev, Hesam Fathi Moghadam, Ali Moharrer, Jingxiao Cai, Nikan Chavoshi, Venkatanathan Varadarajan, Sandeep R Agrawal, Sam Idicula, Tomas Karnagel, Sanjay Jinturkar, et al . 2020. Oracle automl: a fast and predictive automl pipeline.Proceedings of the VLDB Endowment...

    2020

  72. [80]

    Brian Hu Zhang, Blake Lemoine, and Margaret Mitchell. 2018. Mitigating un- wanted biases with adversarial learning. InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society. 335–340

    2018

  73. [81]

    female” and “non-White

    Richard Zhang and Daniel Golovin. 2020. Random Hypervolume Scalarizations for Provable Multi-Objective Black Box Optimization. InProceedings of the 37th International Conference on Machine Learning (Proceedings of Machine Learning Research), Hal Daumé III and Aarti Singh (Eds....

    2020

  74. [2019]

    InProceedings of the 2019 international conference on management of data

    Democratizing data science through interactive curation of ml pipelines. InProceedings of the 2019 international conference on management of data. 1171– 1188

    2019

  75. [2023]

    Multi-objective hyperparameter optimization in machine learning—An overview.ACM Transactions on Evolutionary Learning and Optimization3, 4 (2023), 1–50

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.