REVIEW 4 major objections 5 minor 26 references
VOPy: A Framework for Black-box Vector Optimization
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper introduces VOPy, an open-source Python library for black-box vector optimization that orders solutions by convex-cone partial orders instead of componentwise comparison.
desk verdict VOPy fills a real gap for cone-based vector optimization, but the paper describes rather than demonstrates the core comparison solvers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the combination of an Order interface and a ConfidenceRegion abstraction. The Order interface encodes a vector partial order $\preceq_C$ on $\mathbb{R}^D$ induced by a closed convex cone $C$, so that $\mu \preceq_C \nu$ iff $\nu - \mu \in C$. The ConfidenceRegion abstraction turns each noisy estimate of an objective vector into a set that contains the true value with high probability, then asks whether one such region is dominated by another under $\preceq_C$. RectangularConfidenceRegion handles the componentwise case, while EllipsoidalConfidenceRegion handles dependent objectives and non-polyhedral cones; both reduce the comparison to a convex program that VOPy solves automatically. This mechanism is what lets the library escape the hyperrectangle comparisons of ordinary multi-objective optimization and support general cone-based preferences.
What would settle it
Run VOPy's confidence-region comparisons on a small grid of noisy objective estimates for a 2D polyhedral cone and verify every dominance verdict against an exact brute-force check; the first mismatch would show the comparison mechanism fails.
Extended reading notes
Core claim
The paper presents VOPy as the first software framework for developing black-box vector-optimization algorithms. Its key design claim is that any partial order induced by a closed convex cone can be used throughout the stack: the Order interface defines the cone, the Model interface supplies uncertainty estimates, the Algorithm interface schedules evaluations, and the Problem interface connects datasets or real-world systems. To make dominance checks tractable under arbitrary cones, VOPy provides confidence-region classes—rectangular for componentwise orders and ellipsoidal for dependent or non-polyhedral settings—and solves the required comparisons as convex optimization problems. The paper reports that these components are used by implementations of existing vector-optimization and multi-objective algorithms, including PaVeBa, VOGP, naive elimination, and epsilon-PAL, and that decoupled variants allow partial objective evaluations when costs differ.
Load-bearing premise
The library's usefulness rests on the assumption that its built-in confidence-region comparisons, solved as convex programs, return correct dominance verdicts quickly enough for any supported cone; the paper does not provide equations, experiments, or runtime numbers that demonstrate this.
Editorial extensions
If this is right
- Algorithm developers can build and benchmark new vector-optimization methods against PaVeBa, VOGP, naive elimination, epsilon-PAL, and the decoupled variants without reimplementing infrastructure.
- Practitioners with noisy, batch, or decoupled objective evaluations can apply the same library to real-world problems such as material design, vehicle safety, and streaming-sorting networks using built-in datasets.
- Users with non-polyhedral preferences can define a custom Order derivative and reuse existing models and algorithms, extending the library beyond componentwise optimization.
- The convex-programming confidence-region comparisons replace hyperrectangle comparisons, making dependent-objective models representable exactly instead of approximated by rectangles.
- The authors report that VOPy's early stages contributed to the development of published vector-optimization algorithms, positioning it as a natural common testbed for future research.
Reading between the lines
- If the confidence-region convex programs scale well, the same comparison machinery could be reused outside vector optimization, such as in safe or constrained Bayesian optimization where feasibility regions need to be compared under general cones.
- The authors do not report runtime benchmarks for the convex-programming solvers; a natural next step is a systematic comparison of rectangular versus ellipsoidal confidence regions on the same cone-ordered problems, checking both wall-clock time and statistical efficiency.
- A broader consequence, left implicit, is that a common library with standardized cones and benchmarks could make fairness of comparison across vector-optimization algorithms much easier to audit, since all methods would share the same order logic and uncertainty models.
- The library's support for decoupled algorithms suggests an untested opportunity: using VOPy to study active-learning strategies that query only the cheapest objectives when costs differ, a direction the included entropy-based decoupled algorithm only begins to explore.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces VOPy, an open-source Python library for black-box vector optimization, where multiple objectives are optimized with respect to a partial order induced by a convex cone. The library is organized around four interfaces (Order, Model, Algorithm, Problem), provides confidence-region abstractions with comparisons solved via CVXPY, and includes implementations of several published algorithms (e.g., PaVeBa, VOGP, epsilon-PAL, and an Auer et al. algorithm) as well as decoupled variants. The manuscript describes the architecture and usage but contains no empirical results, no mathematical formulation of the confidence-region comparison problems, and no runtime or benchmark evaluations; the main supporting evidence is a reported code coverage of at least 95% and the availability of a public repository.
Significance. If the implementation is correct and efficient, VOPy would fill a genuine gap: existing multi-objective optimization libraries, such as PyMOO and BoTorch, are built around the componentwise order and do not directly support general convex-cone-induced partial orders. The modular design, the inclusion of state-of-the-art vector-optimization algorithms, and the support for decoupled evaluations are all valuable contributions for algorithm developers. It is also a strength that the source code is public and that the authors follow standard software-engineering practices (PEP8, Flake8, Black, Bandit). However, the paper does not demonstrate that the central implementation is correct or practical: the convex-programming solutions for confidence-region comparisons are never written down, no correctness proofs are given, and no benchmarks or end-to-end experiments are reported. The significance of the contribution therefore rests on unverified claims about an external repository rather than on evidence contained in the manuscript.
major comments (4)
- [Section 2.1] The ConfidenceRegion abstraction is the computational core of the library, but the paper never states the convex program that decides whether one confidence region is preferred over another with respect to a cone C. The text only says that RectangularConfidenceRegion and EllipsoidalConfidenceRegion support these comparisons using CVXPY. Since every algorithm in VOPy must repeatedly invoke this dominance test, the correctness of this encoding is load-bearing. Please provide the explicit optimization formulation for both confidence-region classes, a correctness argument, and a discussion of how the formulation handles polyhedral versus non-polyhedral cones and dependent objectives.
- [Section 2.1] The claim that VOPy 'provides significant speed-ups when possible' is supported only by a reference to a 'Performance notebook' that is not included in the paper and has no link or version identifier. No runtime measurements appear anywhere in the manuscript. As a result, the practicality of contribution (2) cannot be assessed. Include the performance notebook or an equivalent benchmark section with wall-clock times, solver statistics, and comparisons against a baseline, or qualify the performance claim so that it is not verifiable only through the repository.
- [Section 2.2] Reporting code coverage of at least 95% does not establish the correctness of the convex-programming solvers or the reliability of the implemented algorithms. Coverage measures how many code statements are executed, not whether the returned solutions are mathematically correct, whether confidence regions have the claimed coverage probability, or whether the algorithms recover the true Pareto set. The 'demonstrated reliability' stated in Section 4 is therefore not supported. Add end-to-end experiments on synthetic problems with known Pareto sets, verification of the confidence-region comparisons against brute-force oracles, and, where relevant, results on the provided real-world datasets.
- [Section 3] The paper announces a 'novel entropy-based decoupled algorithm' but gives no description, no pseudocode, and no evaluation. If this algorithm is intended as a substantive contribution, it needs a full technical treatment; if it is only an example of the framework's flexibility, that status should be stated explicitly. As written, the contribution list in the introduction and the mention in Section 3 promise more than the manuscript delivers.
minor comments (5)
- [Section 3] The text mentions 'polyhedral approximations of 3D ice-cream cones' without quantifying the approximation error or explaining whether the algorithms' correctness guarantees apply to the approximate cone or to the true cone; please clarify.
- [Section 2.1] The 'Performance notebook' is referenced without a URL or repository path; if it exists, provide a direct link and a commit hash or version tag for reproducibility.
- [Section 2.1] The phrase 'has been checked for vulnerabilities with the tool Bandit and is secure' overstates what a static-analysis tool can establish; rephrase to 'has been scanned with Bandit' or otherwise qualify the security claim.
- [Introduction] The claim that VOPy is the 'first library' for black-box vector optimization would be easier to evaluate if the authors stated the search criteria or the specific features that exclude existing MOO libraries beyond the cone-order distinction.
- [Abstract] The abstract lists 'batch observations' as a supported setting, but the paper does not explain which built-in algorithms or acquisition functions support batch evaluation; please clarify.
Circularity Check
No significant circularity: VOPy is a software framework paper whose algorithmic content reimplements independently published methods; no prediction or derivation reduces to its own inputs.
full rationale
This is a software/engineering paper, not a derivation paper. The central claims are that VOPy is the first library for black-box vector optimization, that it provides modular interfaces for orders/models/algorithms/problems, and that it solves confidence-region comparison convex programs via CVXPY. None of these claims is obtained by fitting a parameter to data and then predicting that same data; there are no fitted values, no empirical prediction claims, and no equations whose equivalence could be exhibited. The self-citations to the authors' prior work (Ararat and Tekin 2023; Karagozlu et al. 2024; Korkmaz et al. 2024) are used as provenance for the implemented algorithms, which are published, peer-reviewed results with independent support; the paper does not invoke those citations as the sole justification for a uniqueness theorem or as a way to smuggle in an ansatz. The statement in Section 4 that 'Its early stages contributed to the development of algorithms now existing in the literature' is a historical note, not a load-bearing reduction. The main unresolved issue is engineering verification: the confidence-region comparison solvers are described only as 'supporting solutions to these comparisons using the CVXPY library' without mathematical formulations or runtime benchmarks, and a 'Performance notebook' is referenced but not included. That is a correctness or completeness concern, not circularity. Accordingly, the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (4)
- domain assumption The comparison of confidence regions with respect to a cone-induced partial order can be reduced to convex optimization problems that CVXPY solves correctly.
- domain assumption The implemented algorithms from prior work (PaVeBa, VOGP, epsilon-PAL) are correct as originally published.
- domain assumption A test coverage above 95 percent implies the library is reliable.
- standard math A closed convex cone C with C intersect -C = {0} defines a valid partial order.
Cite this review
Pith. "Pith review of VOPy: A Framework for Black-box Vector Optimization." pith.science (2026). https://pith.science/paper/YAJH26TW
@misc{pith2026241206604,
author = {Pith},
title = {Pith review of: VOPy: A Framework for Black-box Vector Optimization},
year = {2026},
howpublished = {\url{https://pith.science/paper/YAJH26TW}},
note = {Machine review of arXiv:2412.06604}
}
read the original abstract
We introduce VOPy, an open-source Python library designed to address black-box vector optimization, where multiple objectives must be optimized simultaneously with respect to a partial order induced by a convex cone. VOPy extends beyond traditional multi-objective optimization (MOO) tools by enabling flexible, cone-based ordering of solutions; with an application scope that includes environments with observation noise, discrete or continuous design spaces, limited budgets, and batch observations. VOPy provides a modular architecture, facilitating the integration of existing methods and the development of novel algorithms. We detail VOPy's architecture, usage, and potential to advance research and application in the field of vector optimization. The source code for VOPy is available at https://github.com/Bilkent-CYBORG/VOPy.
Figures
Reference graph
Works this paper leans on
-
[1]
Vector optimization with stochastic bandit feedback
C a g n Ararat and Cem Tekin. Vector optimization with stochastic bandit feedback. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, pages 2165--2190, 2023
work page 2023
-
[2]
Pareto front identification from stochastic bandit feedback
Peter Auer, Chao-Kai Chiang, Ronald Ortner, and Madalina Drugan. Pareto front identification from stochastic bandit feedback. In Proceedings of The 19th International Conference on Artificial Intelligence and Statistics, pages 939--947, 2016
work page 2016
-
[3]
Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy
Maximilian Balandat, Brian Karrer, Daniel R. Jiang, Samuel Daulton, Benjamin Letham, Andrew Gordon Wilson, and Eytan Bakshy. B o T orch: a framework for efficient M onte- C arlo B ayesian optimization. In Advances in Neural Information Processing Systems 33 (NIPS 2020), pages 21524--21538, 2020
work page 2020
-
[4]
Pymoo: Multi-objective optimization in P ython
Julian Blank and Kalyanmoy Deb. Pymoo: Multi-objective optimization in P ython. I E E E Access , 8: 0 89497--89509, 2020
work page 2020
-
[5]
R B F O pt: an open-source library for black-box optimization with costly function evaluations
Alberto Costa and Giacomo Nannicini. R B F O pt: an open-source library for black-box optimization with costly function evaluations. Mathematical Programming Computation, 10: 0 597--629, 2018
work page 2018
-
[6]
A tutorial on multiobjective optimization: fundamentals and evolutionary methods
Michael TM Emmerich and Andr \'e H Deutz. A tutorial on multiobjective optimization: fundamentals and evolutionary methods. Natural Computing, 17: 0 585--609, 2018
work page 2018
-
[7]
Google V izier: a service for black-box optimization
Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and David Sculley. Google V izier: a service for black-box optimization. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1487--1495, 2017
work page 2017
-
[8]
Bayesian optimization for synthetic gene design
Javier Gonzalez, Joseph Longworth, David C James, and Neil D Lawrence. Bayesian optimization for synthetic gene design. arXiv preprint 1505.01627, 2015
arXiv 2015
Show all 26 references
-
[9]
Portfolio allocation for B ayesian optimization
Matthew Hoffman, Eric Brochu, Nando De Freitas, et al. Portfolio allocation for B ayesian optimization. In Proceedings of the 27th Conference on Uncertainty in Artificial Intelligence, pages 327--336, 2011
2011
-
[10]
Bias free multiobjective active learning for materials design and discovery
Kevin Maik Jablonka, Giriprasad Melpatti Jothiappan, Shefang Wang, Berend Smit, and Brian Yoo. Bias free multiobjective active learning for materials design and discovery. Nature Communications, 12 0 (1): 0 2312, 2021
2021
-
[11]
Openbox: a P ython toolkit for generalized black-box optimization
Huaijun Jiang, Yu Shen, Yang Li, Beicheng Xu, Sixian Du, Wentao Zhang, Ce Zhang, and Bin Cui. Openbox: a P ython toolkit for generalized black-box optimization. Journal of Machine Learning Research, 25 0 (120): 0 1--11, 2024
2024
-
[12]
Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnab\' a s P\' o czos, and Eric P. Xing. Neural architecture search with B ayesian optimisation and optimal transport. In Advances in Neural Information Processing Systems 31 (NIPS 2018), pages 2020--2029, 2018
2018
-
[13]
Learning the P areto set under incomplete preferences: Pure exploration in vector bandits
Efe Mert Karag \"o zl \"u , Ya s ar Cahit Y ld r m, C a g n Ararat, and Cem Tekin. Learning the P areto set under incomplete preferences: Pure exploration in vector bandits. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, pages 30...
2024
-
[14]
Vector optimization with G aussian process bandits
İlter Onat Korkmaz, Yaşar Cahit Yıldırım, Çağın Ararat, and Cem Tekin. Vector optimization with G aussian process bandits. arXiv preprint 2412.02484, 2024
2024
-
[15]
Multiobjective optimization for crash safety design of vehicles using stepwise regression model
Xingtao Liao, Qing Li, Xujing Yang, Weigang Zhang, and Wei Li. Multiobjective optimization for crash safety design of vehicles using stepwise regression model. Structural and Multidisciplinary Optimization, 35: 0 561--569, 2008
2008
-
[16]
Crystal structure prediction using ab initio evolutionary techniques: principles and applications
Artem R Oganov and Colin W Glass. Crystal structure prediction using ab initio evolutionary techniques: principles and applications. The Journal of Chemical Physics, 124 0 (24), 2006
2006
-
[17]
Prediction of low-thermal-conductivity compounds with first-principles anharmonic lattice-dynamics calculations and B ayesian optimization
Atsuto Seko, Atsushi Togo, Hiroyuki Hayashi, Koji Tsuda, Laurent Chaput, and Isao Tanaka. Prediction of low-thermal-conductivity compounds with first-principles anharmonic lattice-dynamics calculations and B ayesian optimization. Physical Review Letters, 115 0 (20): 0 205901, 2015
2015
-
[18]
Practical B ayesian optimization of machine learning algorithms
Jasper Snoek, Hugo Larochelle, and Ryan P Adams. Practical B ayesian optimization of machine learning algorithms. Advances in Neural Information Processing Systems 25 (NIPS 2012), pages 2951--2959, 2012
2012
-
[19]
An easy-to-use real-world multi-objective optimization problem suite
Ryoji Tanabe and Hisao Ishibuchi. An easy-to-use real-world multi-objective optimization problem suite. Applied Soft Computing, 89: 0 106078, 2020
2020
-
[20]
Black-box optimization for automated discovery
Kei Terayama, Masato Sumita, Ryo Tamura, and Koji Tsuda. Black-box optimization for automated discovery. Accounts of Chemical Research, 54 0 (6): 0 1334--1346, 2021
2021
-
[21]
Joint entropy search for multi-objective B ayesian optimization
Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi-objective B ayesian optimization. In Advances in Neural Information Processing Systems 35 (NIPS 2022), pages 9922--9938, 2022
2022
-
[22]
C O M B O : An efficient B ayesian optimization library for materials science
Tsuyoshi Ueno, Trevor David Rhone, Zhufeng Hou, Teruyasu Mizoguchi, and Koji Tsuda. C O M B O : An efficient B ayesian optimization library for materials science. Materials Discovery, 4: 0 18--21, 2016
2016
-
[23]
Style guide for Python code
Guido van Rossum, Barry Warsaw, and Nick Coghlan. Style guide for Python code. PEP 8, Python Software Foundation, 2001. URL https://www.python.org/dev/peps/pep-0008/
2001
-
[24]
Hyperparameter optimization for machine learning models based on B ayesian optimization
Jia Wu, Xiu-Yun Chen, Hao Zhang, Li-Dong Xiong, Hang Lei, and Si-Hao Deng. Hyperparameter optimization for machine learning models based on B ayesian optimization. Journal of Electronic Science and Technology, 17 0 (1): 0 26--40, 2019
2019
-
[25]
Computer generation of streaming sorting networks
Marcela Zuluaga, Peter Milder, and Markus P \"u schel. Computer generation of streaming sorting networks. In Proceedings of the 49th Annual Design Automation Conference, pages 1245--1253, 2012
2012
-
[26]
e- P A L : An active learning approach to the multi-objective optimization problem
Marcela Zuluaga, Andreas Krause, and Markus P\" u schel. e- P A L : An active learning approach to the multi-objective optimization problem. Journal of Machine Learning Research, 17 0 (104): 0 1--32, 2016
2016
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.