{"id":"bca90faf-f631-405e-9be5-a28722f5b7a0","arxiv_id":"2412.06604","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"VOPy provides the first modular open-source framework for cone-based black-box vector optimization, with built-in algorithms and confidence-region tools.","lead":"VOPy is a new open-source Python library for black-box vector optimization, where multiple objectives are ranked by a cone-based partial order rather than componentwise comparison. It packages existing algorithms and modular components so researchers can build and benchmark their own vector optimization methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on confidence-region comparison solvers that are described but neither formulated nor benchmarked; without equations or runtime evidence, correctness and practicality of the core contribution are unverified.","rationale":"The reader's weakest assumption correctly identifies the confidence-region comparison solvers as the load-bearing but unverified component of VOPy's central claim. The paper claims to solve the convex optimization problems for these comparisons, yet provides no equations, no derivation, and no experiments to substantiate either correctness or computational efficiency. Without those, the 'first library' claim cannot be fully supported, because a library whose core comparisons are unreliable or too slow is not a usable framework for black-box vector optimization. The reader's verdict of CONDITIONAL is appropriate: the condition should be that the authors provide explicit convex formulations and either unit-level correctness tests or independent benchmarks. I found no stronger internal inconsistency that would justify a REJECT or UNVERDICTED stance; the code is publicly available, so independent verification is feasible. The concern is real, but it is addressable by concrete evidence rather than being a fundamental logical flaw. Therefore, I agree with the reader's assessment and see no need to change the verdict.","tokens_in":5229,"tokens_out":3947,"duration_ms":44159,"concrete_test":"Run the library's existing test suite and add a randomized cross-check that compares the CVXPY-based dominance decisions of RectangularConfidenceRegion and EllipsoidalConfidenceRegion against a brute-force grid discretization of the two regions for a 3-dimensional polyhedral cone (e.g., the componentwise order and a 5-facet cone); record both the number of decision mismatches and the wall-clock time per comparison as a function of dimension (D = 2, 4, 8) and batch size. If any mismatch occurs or runtime grows super-polynomially in D, the solver correctness or efficiency assumption fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that VOPy is the first library for black-box vector optimization, and contribution (2) asserts that the authors 'provide solutions to the convex optimization problems that arise while performing confidence region comparisons' with respect to a cone. The only technical description appears in Section 2.1, where RectangularConfidenceRegion and EllipsoidalConfidenceRegion are introduced as derivatives of ConfidenceRegion, both 'supporting solutions to these comparisons using the CVXPY library.' No mathematical formulation is given for how a dominance test is encoded as a convex program, no proof of correctness is offered, and no runtime measurements are reported. The text even references a 'Performance notebook' that is not included in the paper. Because every algorithm built on VOPy must repeatedly invoke these comparisons to decide whether one confidence region is preferred over another, any error or excessive slowdown in these solvers would invalidate the framework's core promise. The absence of derivations and benchmarks makes it impossible to assess whether the central claim is true, even though the public repository is available for independent verification.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces VOPy, an open-source Python library for black-box vector optimization, where multiple objectives are optimized with respect to a partial order induced by a convex cone. The library is organized around four interfaces (Order, Model, Algorithm, Problem), provides confidence-region abstractions with comparisons solved via CVXPY, and includes implementations of several published algorithms (e.g., PaVeBa, VOGP, epsilon-PAL, and an Auer et al. algorithm) as well as decoupled variants. The manuscript describes the architecture and usage but contains no empirical results, no mathematical formulation of the confidence-region comparison problems, and no runtime or benchmark evaluations; the main supporting evidence is a reported code coverage of at least 95% and the availability of a public repository.","tokens_in":5403,"tokens_out":4856,"duration_ms":55354,"significance":"If the implementation is correct and efficient, VOPy would fill a genuine gap: existing multi-objective optimization libraries, such as PyMOO and BoTorch, are built around the componentwise order and do not directly support general convex-cone-induced partial orders. The modular design, the inclusion of state-of-the-art vector-optimization algorithms, and the support for decoupled evaluations are all valuable contributions for algorithm developers. It is also a strength that the source code is public and that the authors follow standard software-engineering practices (PEP8, Flake8, Black, Bandit). However, the paper does not demonstrate that the central implementation is correct or practical: the convex-programming solutions for confidence-region comparisons are never written down, no correctness proofs are given, and no benchmarks or end-to-end experiments are reported. The significance of the contribution therefore rests on unverified claims about an external repository rather than on evidence contained in the manuscript.","major_comments":[{"comment":"The ConfidenceRegion abstraction is the computational core of the library, but the paper never states the convex program that decides whether one confidence region is preferred over another with respect to a cone C. The text only says that RectangularConfidenceRegion and EllipsoidalConfidenceRegion support these comparisons using CVXPY. Since every algorithm in VOPy must repeatedly invoke this dominance test, the correctness of this encoding is load-bearing. Please provide the explicit optimization formulation for both confidence-region classes, a correctness argument, and a discussion of how the formulation handles polyhedral versus non-polyhedral cones and dependent objectives.","section":"Section 2.1"},{"comment":"The claim that VOPy 'provides significant speed-ups when possible' is supported only by a reference to a 'Performance notebook' that is not included in the paper and has no link or version identifier. No runtime measurements appear anywhere in the manuscript. As a result, the practicality of contribution (2) cannot be assessed. Include the performance notebook or an equivalent benchmark section with wall-clock times, solver statistics, and comparisons against a baseline, or qualify the performance claim so that it is not verifiable only through the repository.","section":"Section 2.1"},{"comment":"Reporting code coverage of at least 95% does not establish the correctness of the convex-programming solvers or the reliability of the implemented algorithms. Coverage measures how many code statements are executed, not whether the returned solutions are mathematically correct, whether confidence regions have the claimed coverage probability, or whether the algorithms recover the true Pareto set. The 'demonstrated reliability' stated in Section 4 is therefore not supported. Add end-to-end experiments on synthetic problems with known Pareto sets, verification of the confidence-region comparisons against brute-force oracles, and, where relevant, results on the provided real-world datasets.","section":"Section 2.2"},{"comment":"The paper announces a 'novel entropy-based decoupled algorithm' but gives no description, no pseudocode, and no evaluation. If this algorithm is intended as a substantive contribution, it needs a full technical treatment; if it is only an example of the framework's flexibility, that status should be stated explicitly. As written, the contribution list in the introduction and the mention in Section 3 promise more than the manuscript delivers.","section":"Section 3"}],"minor_comments":[{"comment":"The text mentions 'polyhedral approximations of 3D ice-cream cones' without quantifying the approximation error or explaining whether the algorithms' correctness guarantees apply to the approximate cone or to the true cone; please clarify.","section":"Section 3"},{"comment":"The 'Performance notebook' is referenced without a URL or repository path; if it exists, provide a direct link and a commit hash or version tag for reproducibility.","section":"Section 2.1"},{"comment":"The phrase 'has been checked for vulnerabilities with the tool Bandit and is secure' overstates what a static-analysis tool can establish; rephrase to 'has been scanned with Bandit' or otherwise qualify the security claim.","section":"Section 2.1"},{"comment":"The claim that VOPy is the 'first library' for black-box vector optimization would be easier to evaluate if the authors stated the search criteria or the specific features that exclude existing MOO libraries beyond the cone-order distinction.","section":"Introduction"},{"comment":"The abstract lists 'batch observations' as a supported setting, but the paper does not explain which built-in algorithms or acquisition functions support batch evaluation; please clarify.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript currently reads as an extended abstract or project announcement rather than a self-contained software-paper evaluation. The central gap is missing technical formulation and missing empirical validation, not an error that is impossible to fix within the paper's scope. I therefore recommend major revision rather than rejection. If the venue's software-track expectations do not permit adding benchmarks of the type requested, the paper may be better suited to a platform that accepts external repository assessment; however, the missing performance notebook and the unsupported 'first library' claim should be resolved in any case."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the quick take: VOPy fills a genuine gap — I don't know another open-source library that supports black-box optimization under general convex cone partial orders. The modular design (Order, Model, Algorithm, Problem) is clean and borrows sensibly from BoTorch. The authors also implement several published algorithms (PaVeBa, VOGP, epsilon-PAL) and add two decoupled variants. The code is public, so the artifact is checkable. That's real work and worth credit.\n\nThe soft spots, in proportion. The manuscript itself is thin. Contribution (2) — 'we provide solutions to the convex optimization problems that arise in confidence-region comparisons' — is never actually formulated. No equations, no correctness argument, no runtime numbers. The paper mentions a 'Performance notebook' that isn't included. That matters, because every algorithm in the library leans on those comparisons: if the CVXPY routines are slow or wrong, the framework's promise doesn't hold. The reader's concern here is on point. The 95% test coverage is nice, but it doesn't establish that the convex programs are correct or efficient.\n\nThis is a software paper, so I don't treat the lack of theorems as fatal. But the lack of any empirical evidence in the paper is a real weakness. The fix is straightforward: derive the dominance tests for at least the ellipsoidal case, and include a reproducible benchmark comparing against componentwise MOO baselines on a few standard problems. Even a link to a working demo notebook in the supplement would raise my confidence substantially.\n\nI'd send this to peer review — the gap is real and the library could accelerate research in vector optimization. But I'd make acceptance conditional on the authors adding the missing formulation and at least a minimal experimental section. I wouldn't cite it in my own work yet, because the central claim is unverified from the paper alone. For a reading group, it's a maybe: worth a look for anyone interested in VO, but expect to dig into the GitHub repository rather than the manuscript.","headline":"VOPy fills a real gap for cone-based vector optimization, but the paper describes rather than demonstrates the core comparison solvers.","tokens_in":5940,"tokens_out":2488,"would_cite":false,"duration_ms":24780,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces VOPy, an open-source Python library for black-box vector optimization that orders solutions by convex-cone partial orders instead of componentwise comparison.","keywords":["vector optimization","multi-objective optimization","Bayesian optimization","multi-armed bandit","black-box optimization","convex cone partial order","confidence region comparison","open-source software"],"falsifier":"Run VOPy's confidence-region comparisons on a small grid of noisy objective estimates for a 2D polyhedral cone and verify every dominance verdict against an exact brute-force check; the first mismatch would show the comparison mechanism fails.","tokens_in":5100,"feed_emoji":"🎯","tokens_out":8408,"duration_ms":84962,"temperature":0.7,"pith_summary":"VOPy is an open-source Python library for black-box vector optimization, the setting where several objectives are optimized together under a partial order defined by a convex cone rather than the componentwise order of standard multi-objective optimization. The paper's central claim is that VOPy is the first library built for this task, covering noisy observations, discrete or continuous design spaces, limited budgets, batch observations, and decoupled evaluations. The library organizes around four interfaces—Order, Model, Algorithm, and Problem—so that users can apply existing methods, assemble new algorithms from standard components, or drop in custom orders and models. It also implements a confidence-region comparison mechanism that decides dominance between uncertain solutions by solving convex programs, extending comparisons beyond the hyperrectangles used in ordinary multi-objective tools. A sympathetic reader would care because this fills a missing infrastructure layer for a growing family of vector-optimization algorithms.","feed_headline":"VOPy: first library for black-box vector optimization","feed_subtitle":"Generalizes multi-objective optimization to arbitrary cone orders, covering noisy, batch, and decoupled evaluations in one framework.","key_machinery":"The load-bearing mechanism is the combination of an Order interface and a ConfidenceRegion abstraction. The Order interface encodes a vector partial order $\\preceq_C$ on $\\mathbb{R}^D$ induced by a closed convex cone $C$, so that $\\mu \\preceq_C \\nu$ iff $\\nu - \\mu \\in C$. The ConfidenceRegion abstraction turns each noisy estimate of an objective vector into a set that contains the true value with high probability, then asks whether one such region is dominated by another under $\\preceq_C$. RectangularConfidenceRegion handles the componentwise case, while EllipsoidalConfidenceRegion handles dependent objectives and non-polyhedral cones; both reduce the comparison to a convex program that VOPy solves automatically. This mechanism is what lets the library escape the hyperrectangle comparisons of ordinary multi-objective optimization and support general cone-based preferences.","core_discovery":"The paper presents VOPy as the first software framework for developing black-box vector-optimization algorithms. Its key design claim is that any partial order induced by a closed convex cone can be used throughout the stack: the Order interface defines the cone, the Model interface supplies uncertainty estimates, the Algorithm interface schedules evaluations, and the Problem interface connects datasets or real-world systems. To make dominance checks tractable under arbitrary cones, VOPy provides confidence-region classes—rectangular for componentwise orders and ellipsoidal for dependent or non-polyhedral settings—and solves the required comparisons as convex optimization problems. The paper reports that these components are used by implementations of existing vector-optimization and multi-objective algorithms, including PaVeBa, VOGP, naive elimination, and epsilon-PAL, and that decoupled variants allow partial objective evaluations when costs differ.","pith_inferences":["If the confidence-region convex programs scale well, the same comparison machinery could be reused outside vector optimization, such as in safe or constrained Bayesian optimization where feasibility regions need to be compared under general cones.","The authors do not report runtime benchmarks for the convex-programming solvers; a natural next step is a systematic comparison of rectangular versus ellipsoidal confidence regions on the same cone-ordered problems, checking both wall-clock time and statistical efficiency.","A broader consequence, left implicit, is that a common library with standardized cones and benchmarks could make fairness of comparison across vector-optimization algorithms much easier to audit, since all methods would share the same order logic and uncertainty models.","The library's support for decoupled algorithms suggests an untested opportunity: using VOPy to study active-learning strategies that query only the cheapest objectives when costs differ, a direction the included entropy-based decoupled algorithm only begins to explore."],"forward_implications":["Algorithm developers can build and benchmark new vector-optimization methods against PaVeBa, VOGP, naive elimination, epsilon-PAL, and the decoupled variants without reimplementing infrastructure.","Practitioners with noisy, batch, or decoupled objective evaluations can apply the same library to real-world problems such as material design, vehicle safety, and streaming-sorting networks using built-in datasets.","Users with non-polyhedral preferences can define a custom Order derivative and reuse existing models and algorithms, extending the library beyond componentwise optimization.","The convex-programming confidence-region comparisons replace hyperrectangle comparisons, making dependent-objective models representable exactly instead of approximated by rectangles.","The authors report that VOPy's early stages contributed to the development of published vector-optimization algorithms, positioning it as a natural common testbed for future research."],"supporting_citations":[{"why":"Provides the modular Bayesian-optimization design template that VOPy adapts and is the MOO library whose generality VOPy extends.","marker":"BoTorch (Balandat et al., 2020)"},{"why":"Represents the comprehensive multi-objective optimization suite that lacks cone-based orderings, establishing the gap VOPy fills.","marker":"PyMOO (Blank and Deb, 2020)"},{"why":"Supplies the stochastic-bandit vector-optimization formulation and the Naive Elimination algorithm that VOPy implements.","marker":"Ararat and Tekin (2023)"},{"why":"Supplies PaVeBa and PaVeBa-GP, the pure-exploration algorithms and epsilon-F1 metric implemented in VOPy.","marker":"Karagözlü et al. (2024)"},{"why":"Supplies VOGP and its adaptive-discretization extension for continuous domains, which VOPy implements.","marker":"Korkmaz et al. (2024)"},{"why":"Supplies the epsilon-PAL active-learning algorithm and the SNW dataset that VOPy ports.","marker":"Zuluaga et al. (2016)"},{"why":"Supplies the Pareto-front-identification algorithm with no official code, which VOPy implements from the literature.","marker":"Auer et al. (2016)"}],"fun_headline_variants":["VOPy: first Python framework for black-box vector optimization","VOPy: arbitrary cone orders for black-box optimization","VOPy: open-source library for black-box vector optimization","VOPy: handles noise, batches, and discrete spaces in vector optimization","VOPy: first framework for cone-ordered black-box optimization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The library's usefulness rests on the assumption that its built-in confidence-region comparisons, solved as convex programs, return correct dominance verdicts quickly enough for any supported cone; the paper does not provide equations, experiments, or runtime numbers that demonstrate this.","fun_headline_variants_meta":{"raw":{"variants":["VOPy: first Python framework for black-box vector optimization","VOPy: arbitrary cone orders for black-box optimization","VOPy: open-source library for black-box vector optimization","VOPy: handles noise, batches, and discrete spaces in vector optimization","VOPy: first framework for cone-ordered black-box optimization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000589,"raw_usage":{"total_tokens":2711,"prompt_tokens":840,"completion_tokens":1871,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":1782}},"tokens_in":456,"tokens_out":1871,"duration_ms":12513,"temperature":1.0,"reasoning_tokens":1782,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:28:12.588982+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run VOPy's confidence-region comparisons on a small grid of noisy objective estimates for a 2D polyhedral cone and verify every dominance verdict against an exact brute-force check; the first mismatch would show the comparison mechanism fails.","supporting_citations":[{"cited_title":"Vector optimization with stochastic bandit feedback","cited_arxiv_id":null,"evidence_quote":"Supplies the stochastic-bandit vector-optimization formulation and the Naive Elimination algorithm that VOPy implements."},{"cited_title":"Vector optimization with G aussian process bandits","cited_arxiv_id":null,"evidence_quote":"Supplies VOGP and its adaptive-discretization extension for continuous domains, which VOPy implements."},{"cited_title":"e- P A L : An active learning approach to the multi-objective optimization problem","cited_arxiv_id":null,"evidence_quote":"Supplies the epsilon-PAL active-learning algorithm and the SNW dataset that VOPy ports."},{"cited_title":"Pareto front identification from stochastic bandit feedback","cited_arxiv_id":null,"evidence_quote":"Supplies the Pareto-front-identification algorithm with no official code, which VOPy implements from the literature."}],"review_version":1}