{"id":"61dad496-d59e-4cbd-804c-e4b27888546b","arxiv_id":"2501.03398","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"raxpy automates space-filling experimental design for Python functions with optional and hierarchical inputs, and its FSS-LHD-VP-MP algorithm ranks best on the paper's internal metrics.","lead":"raxpy is a new open-source Python package that designs space-filling experiments for functions whose inputs can be optional or hierarchical, using Python type annotations to describe the input space. It proposes several design algorithms and reports that one of them, FSS-LHD-VP-MP, produces better coverage on synthetic test spaces than random or basic Latin hypercube approaches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation (7) treats optional-child null status as independent of parent activation, so FSS target allocations are misspecified for hierarchical spaces; the top ranking may be an artifact of those targets.","rationale":"The paper's strongest claim is the Table 1 ranking, especially FSS-LHD-VP-MP at rank 1 on M_idis. The reader identified the arbitrary null_portion heuristic as the weakest assumption; I found a sharper issue at the same location. Equation (7) is the allocation target that the FSS algorithms optimize and that M_wdsr uses for evaluation. Hierarchical constraints mean child activation is not independent of parent activation: for non-Optional dataclass children the conditional probability is 1, but Eq. (7) multiplies independent per-dimension probabilities. As a result, the product formula is not the probability of a feasible full-sub-space, and its values over feasible FSSs sum below 1, leaving probability mass on infeasible combinations. This is an internal modeling inconsistency rather than a disagreement with outside consensus, and it is more load-bearing than the 1/4 default because it would affect any user-supplied null_portion. The software contribution and the algorithmic ideas are real, and the package is available open-source, so the issue is repairable. But until the targets are corrected or the comparison is rerun with them, 'improved' should be read as conditional. My proposed test is a concrete check: recompute the sum of Eq. (7) over feasible FSSs and rerun the ranking with conditional targets. I partially agree with the reader: we both point to Section 3.3, but the independence flaw is the more fundamental problem.","tokens_in":15832,"tokens_out":10964,"duration_ms":109676,"concrete_test":"Independent re-derivation check: for Ssimple, enumerate PP(Doptional) and sum the Eq. (7) products over all feasible Do; if the sum is not 1 (allowing rounding), Eq. (7) is not a proper allocation distribution. Then re-run the Section 4.2 ranking with corrected conditional targets: non-Optional dataclass children get P(non-null | parent active) = 1, optional children keep their specified null_portion, and the resulting feasible-FSS probabilities are renormalized. Compare FSS-LHD-VP-MP's rank on M_idis and M_wdsr; if it is no longer rank 1 on M_idis, or its M_wdsr rank changes materially, the central improvement claim depends on the misspecified target.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 3.3's Eq. (7) sets the target fraction of design points for full-sub-space Do as \\prod_{k in Do}(1 - alpha_bar_k), a product of per-dimension non-null probabilities. This assumes the null events of child dimensions are independent of their parent's activation, which is false for the dataclass hierarchies used in Section 4. In Ssimple, x4 is a non-Optional float field of HierarchicalFactorOne: whenever parent x3 is active, x4 is certain to be non-null, so P(A4=1 | A3=1) = 1, not 1 - alpha_bar_4. The product target for the full-sub-space {1,2,3,4} is (0.75)^3 = 0.422 rather than 0.562. For the whole family of feasible FSSs, the Eq. (7) percentages do not sum to 1, because probability mass is assigned to infeasible parent-active/child-null states. FSS-LHD-VP-MP is explicitly constructed to match these targets, and the FSS-allocation-difference criterion in Section 4.2 rewards matching them; the M_wdsr metric uses the same weights. The headline 'improved' ranking is therefore partly a comparison against a target that does not describe the actual constrained input space. The default null_portion = 1/4 in Section 3.3, while arbitrary, is secondary: the misspecification persists for any user-supplied null portions because of the independence assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"raxpy, an open-source Python package, introspects annotated function signatures to define input spaces and automatically constructs space-filling designs for optional and hierarchical dimensions. The paper defines seven extended evaluation criteria (M_ocov, M_idis, M_adis, M_wdsr, M_sdsr, a MaxPro variant, and an FSS allocation-difference measure), proposes five algorithms (FSS-LHD, FSS-Random, FSS-LHD-VP, TT-LHD, P-LHD), and adds a simulated-annealing MaxPro post-optimization. It then compares twelve algorithm variants on four synthetic input spaces (basic, simple, modest, complex) at three design sizes with 30 replications each. Table 1 reports average rankings and the authors conclude that FSS-LHD-VP-MP is the best for M_idis and among the top three on all four metrics, supporting the claim that the proposed algorithms improve space-filling designs for optional and hierarchical inputs.","tokens_in":16243,"tokens_out":10571,"duration_ms":90972,"significance":"The software contribution is genuine: raxpy is MIT-licensed, documented, and lowers the barrier to space-filling experiments on Python-callable functions; the paper also identifies a real gap in the literature and proposes a concrete set of metric extensions. However, the numerical evidence for the headline claim is currently internal: the FSS-based algorithms are constructed to match target allocations computed in Section 3.3, and the evaluation metrics reward that match. In hierarchical spaces the target calculation is misspecified (major comment 1), so the reported rankings do not yet establish improved space-filling performance. The significance would be high if the comparisons are re-run with correctly conditioned targets and validated against independent criteria.","major_comments":[{"comment":"Equation (7) cannot define a probability distribution over full sub-spaces as written. It multiplies only over dimensions in Do and omits null-portion factors for optional dimensions outside Do. In Sbasic, with alpha_bar=0.25, the raw targets for {1}, {1,2}, {1,3}, {1,2,3} would be 1, 0.75, 0.75, and 0.5625, which neither sum to 1 nor match the independent probabilities 0.0625, 0.1875, 0.1875, and 0.5625. Additionally, the product treats child null status as independent of parent activation; in Ssimple, x4 is a non-Optional field of HierarchicalFactorOne, so whenever parent x3 is active, x4 is non-null with probability 1, not 1-alpha_bar_4, and the target for {1,2,3,4} should therefore be 0.75*0.75*1=0.562 rather than 0.75^3=0.422. Since FSS-based algorithms are explicitly designed to match these targets and M_wdsr plus the FSS allocation-difference metric reward that match, the 'improved' ranking in Table 1 is partly an artifact of the misspecified target. Please recompute targets as probabilities over the feasible FSS tree (using the constraint set P), or require user-supplied allocations, and re-run the comparisons.","section":"Section 3.3, Eq. (7)"},{"comment":"The M_wdcr column in Table 1 (presumably M_wdsr from Section 3.1) is compared across algorithms with different FSS point allocations, which violates the restriction stated when M_wdsr is introduced: 'We only employ this metric for design comparisons given designs with the same FSS point allocations.' The caption itself marks Random*, TT-LHD*, and P-LHD* as having 'common divergence from the target number of points for each full-sub-space causing measurement biases,' yet their M_wdcr values are still reported and ranked. FSS-based algorithms match the target by construction, so this column partially measures agreement with a heuristic target rather than an independent space-filling property. If M_wdsr is used across algorithms, the comparison should be restricted to designs with identical FSS allocations or adjusted by conditioning on the realized FSS.","section":"Section 4.2, Table 1"},{"comment":"Table 1 reports average ranks over 30 replications without confidence intervals, standard errors, or significance tests. With 12 algorithms, four input spaces, and three sample sizes, differences such as average ranks of 1.0 versus 2.33 can easily be within replication noise, and Figure 5 shows visible variability. In addition, the default null_portion=1/4 heuristic in Section 3.3 sets the target allocations, and no sensitivity analysis is reported for other choices of null_portion or for user-supplied allocations. Please add uncertainty quantification and a robustness check across null_portion values, or temper the conclusion to claim a comparison only for the particular default configuration.","section":"Section 4.1"}],"minor_comments":[{"comment":"In the first case of dist_k, the expression reads |xik - xik|; it should be |xik - xjk|, otherwise the term is identically zero.","section":"Equation (3)"},{"comment":"Metric names are inconsistent: Section 3.1 defines M_sdsr and M_wdsr, while Section 4.1 and Table 1 use M_sdcr and M_wdcr. Please unify the notation.","section":"Section 4.1 and Table 1"},{"comment":"The caption refers to 'FSS-LHP-VP-MP'; this should be 'FSS-LHD-VP-MP'.","section":"Figure 5 caption"},{"comment":"The sentence 'one unit of the complexity represents the null region' is ambiguous; near the end of the paragraph it would be clearer to state explicitly that for real optional dimensions the default is null_portion = 1/4 and that this value drives the FSS allocations in Eq. (7).","section":"Section 3.3"},{"comment":"The bibliography entry for Lindauer et al. (2019) lists the tool as 'Boah', but the text refers to it as the ConfigSpace package; please correct the title.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The core issue is that the evaluation loop is closed: the same heuristic target allocation is used to construct, weight, and judge the designs. This is fixable by recomputing targets from the hierarchy and by adding external validation, but until then the empirical ranking should not be taken at face value. The paper is otherwise a reasonable software-plus-methods contribution and may be suitable after major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is worth a look, but not for the reason the abstract says. The package raxpy addresses a real, underserved niche: space-filling experimental designs for Python functions with optional and hierarchical inputs. The annotation-based space definition is a nice idea, the code is open source under MIT, and the paper honestly reviews the existing landscape. That part is solid.\n\nWhat is actually new: the combination of LHD-style value pooling with MaxPro optimization for hierarchical spaces, and the explicit handling of nullable dimensions in the metrics. The authors correctly identify that existing tools like ConfigSpace and scipy don't provide space-filling designs for these structured spaces. The software contribution is real and likely useful to people running simulation studies with conditional parameters.\n\nNow the soft spot, and it is a load-bearing one. Equation (7) sets target point allocations as a product of per-dimension non-null probabilities, which assumes the null status of child dimensions is independent of their parent's activation. In the paper's own Ssimple example, x4 is a non-optional field of HierarchicalFactorOne, so it is automatically active whenever x3 is active. The product formula assigns probability mass to infeasible sub-spaces like {1,2,3} without x4, and it over-allocates to feasible sub-spaces by ignoring the conditional structure. The FSS-based algorithms are then explicitly designed to match those misspecified targets, and the evaluation rewards them for doing so. The asterisked algorithms in Table 1 are penalized partly for respecting the actual hierarchy rather than the incorrect target. This is not a minor quibble; it directly undermines the headline claim that the proposed algorithms produce 'improved' designs.\n\nThe secondary issues are the arbitrary null_portion heuristic (Chapter 3.3), which sets complexity to 3 for real dimensions and thus fixes the null region at a quarter of the dimension, and the lack of any confidence intervals or significance tests on the rankings. The evaluation is entirely internal, using self-defined metrics on synthetic spaces. These would be fixable with a revised evaluation, but the independence assumption in Eq. (7) requires a more fundamental rethinking of the target allocation.\n\nWho is this for? Practitioners who want to run space-filling experiments on Python functions with optional and hierarchical inputs will find the package useful. The paper deserves peer review because the software and the identified gap are real, but the empirical claims need major revision. I would recommend the editor send it out, with the expectation that the authors rework the allocation targets to respect the hierarchy and add proper statistical comparisons.","headline":"A genuinely useful open-source tool for a real gap, but the algorithm comparison is built on a target allocation that ignores hierarchical dependencies, so the 'improved' claim needs revision before it stands.","tokens_in":16620,"tokens_out":4195,"would_cite":false,"duration_ms":37540,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05"],"pacs":[],"model":"deepseek-v4-flash","headline":"raxpy introduces five algorithms for space-filling designs over optional and hierarchical inputs, and its FSS-LHD-VP-MP variant ranks first on minimum interpoint distance across all tested spaces and sizes.","keywords":["space-filling design","computer experiments","optional dimensions","hierarchical dimensions","maximum projection","Latin hypercube design","Python annotations","design of experiments"],"falsifier":"Run the same 12-algorithm comparison on spaces with real-valued optional dimensions whose null_portion is set to 0.05 or 0.5 instead of the default 0.25; if FSS-LHD-VP-MP no longer ranks near the top on minimum interpoint distance or star discrepancy, the claimed improvement depends on the heuristic rather than on the structural use of full-sub-spaces.","tokens_in":15679,"feed_emoji":"🧪","tokens_out":7013,"duration_ms":63066,"temperature":0.7,"pith_summary":"The paper introduces raxpy, a Python package that turns annotated Python functions into space-filling experimental designs, and proposes five algorithms for input spaces where some dimensions are optional and others are hierarchical. The central claim is that these algorithms create improved space-filling designs for such spaces, with the FSS-LHD-VP-MP variant ranking first on minimum interpoint distance and in the top three on every metric considered in the numerical study. A sympathetic reader should care because computer simulations, hyperparameter searches, and similar experiments routinely have optional or hierarchical inputs, while standard space-filling methods assume every input is always active. The package also automates the mapping from function signature to design to execution, removing a manual step in exploration experiments.","feed_headline":"Raxpy algorithm wins space-filling tests for optional inputs","feed_subtitle":"Raxpy automates space-filling designs for optional and hierarchical Python inputs; FSS-LHD-VP-MP ranks first on minimum interpoint distance.","key_machinery":"The load-bearing machinery is the full-sub-space decomposition: feasible combinations of active dimensions are enumerated, target point allocations are computed from per-dimension null_portion values via the product formula in Eq. (7), and designs are built and evaluated sub-space by sub-space. A null-aware distance metric treats a null value as distance 1 from any active value and distance 0 from another null value, which lets criteria like minimum interpoint distance and MaxPro be applied to spaces with optional dimensions. The proposed algorithms layer these ideas: FSS-LHD and its value-pool variant build separate designs per full-sub-space, while TT-LHD and P-LHD flatten or traverse the hierarchy, and a simulated-annealing MaxPro variant (the -MP suffix) re-optimizes columns while preserving the design's full-sub-space allocation structure.","core_discovery":"The paper's discovery is that optional and hierarchical input spaces can be handled by decomposing the design into full-sub-spaces (FSS), allocating the trial points to those sub-spaces according to null-portion heuristics, filling each sub-space with Latin hypercube designs, pooling values, and optionally post-processing with simulated-annealing MaxPro optimization. Across four input spaces of increasing complexity and three design sizes, with 30 replications per combination, FSS-LHD-VP-MP achieves an average rank of 1.0 on minimum interpoint distance, 2.5 on average single-dimension projection distance, 2.5 on star discrepancy, and 4.25 on weighted full-sub-space discrepancy. The authors argue that algorithms which allocate points by full-sub-space match the target allocations derived from null-portion attributes, whereas random, tree-traversal, and whole-projection designs often diverge from those targets.","pith_inferences":["Beyond the paper: the same full-sub-space decomposition could be embedded in sequential or Bayesian optimization over hierarchical hyperparameter spaces, using a prior over configurations rather than a null_portion heuristic to set target allocations.","Beyond the paper: if the evaluation criterion were downstream surrogate model accuracy instead of geometric space-filling properties, the ranking of the proposed algorithms could change; this is a testable follow-up that the paper does not run.","Beyond the paper: when a user knows the true probability that an optional input is null, explicitly specifying null_portion or direct full-sub-space allocations should replace the default heuristic, and the algorithm ranking should be re-checked under those user-specified targets."],"forward_implications":["raxpy can take a Python function annotated with typing.Optional, typing.Union, and dataclasses and automatically produce and execute a full space-filling design with no separate space-specification file.","FSS-LHD-VP-MP gives the best average minimum interpoint distance across all tested input spaces and design sizes, and it ranks in the top three algorithms on average projection distance, star discrepancy, and weighted discrepancy.","Adding MaxPro optimization raises minimum interpoint distance and improves star discrepancy, but it can worsen the weighted full-sub-space discrepancy, so users who expect to focus on a single sub-space after exploration may prefer not to apply it.","Algorithms that build designs full-sub-space by full-sub-space match the target point allocations derived from null_portion attributes, while random, tree-traversal, and whole-projection designs often miss those targets and bias the resulting evaluation.","Because raxpy supports parallelism and distributed execution, the generated designs can be run on expensive external simulations, command-line programs, and web services as well as on ordinary Python functions."],"supporting_citations":[{"why":"Supplies the minimax and maximin distance criterion that becomes the minimum interpoint distance metric M_idis.","marker":"[Johnson et al., 1990]"},{"why":"Provides the MaxPro approach for designs with multiple factor types that the paper extends to optional and hierarchical dimensions.","marker":"[Joseph et al., 2020]"},{"why":"Provides the MaxPro package and algorithm used as the simulated-annealing post-optimization method.","marker":"[Shan Ba and V . Roshan Joseph, 2015]"},{"why":"Supplies the scipy centered-discrepancy LatinHypercube algorithm used as the traditional space-filling design method inside the proposed algorithms.","marker":"[Virtanen et al., 2020]"},{"why":"Motivates Latin hypercube optimization and subprojection properties that the paper's projection metrics build on.","marker":"[Damblin et al., 2013]"},{"why":"Defines discrepancy measures, including star discrepancy, that the paper extends to spaces with null regions.","marker":"[Zhou et al., 2013]"},{"why":"Addresses fast flexible space-filling designs with nominal factors for nonrectangular regions, related to the non-rectangular structure introduced by optional dimensions.","marker":"[Lekivetz and Jones, 2019]"},{"why":"Motivates hierarchical input spaces through hyperparameter optimization, one of the application domains the paper targets.","marker":"[Bergstra et al., 2011]"}],"fun_headline_variants":["Raxpy automates space-filling for optional and hierarchical inputs","Raxpy algorithm wins space-filling for optional inputs","Raxpy's FSS-LHD-MP ranks first on minimum distance","New Python package raxpy improves space-filling designs","Raxpy: automated space-filling for optional and hierarchical inputs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is the default null_portion heuristic: every real-valued optional dimension is assumed to have complexity 3, so its null region occupies one quarter of that dimension; if the actual probability that an optional input is absent differs, the target allocations change and the reported algorithm rankings may not transfer.","fun_headline_variants_meta":{"raw":{"variants":["Raxpy automates space-filling for optional and hierarchical inputs","Raxpy algorithm wins space-filling for optional inputs","Raxpy's FSS-LHD-MP ranks first on minimum distance","New Python package raxpy improves space-filling designs","Raxpy: automated space-filling for optional and hierarchical inputs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000518,"raw_usage":{"total_tokens":2470,"prompt_tokens":862,"completion_tokens":1608,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1519}},"tokens_in":478,"tokens_out":1608,"duration_ms":11947,"temperature":1.0,"reasoning_tokens":1519,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:51:58.104753+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 12-algorithm comparison on spaces with real-valued optional dimensions whose null_portion is set to 0.05 or 0.5 instead of the default 0.25; if FSS-LHD-VP-MP no longer ranks near the top on minimum interpoint distance or star discrepancy, the claimed improvement depends on the heuristic rather than on the structural use of full-sub-spaces.","supporting_citations":[{"cited_title":"Roshan Joseph","cited_arxiv_id":null,"evidence_quote":"Provides the MaxPro package and algorithm used as the simulated-annealing post-optimization method."},{"cited_title":"Numerical studies of space-filling designs: optimization of Latin Hypercube Samples and subprojection properties","cited_arxiv_id":null,"evidence_quote":"Motivates Latin hypercube optimization and subprojection properties that the paper's projection metrics build on."},{"cited_title":"Mixture discrepancy for quasi-random point sets","cited_arxiv_id":null,"evidence_quote":"Defines discrepancy measures, including star discrepancy, that the paper extends to spaces with null regions."},{"cited_title":"Fast flexible space‐filling designs with nominal factors for nonrectangular regions","cited_arxiv_id":null,"evidence_quote":"Addresses fast flexible space-filling designs with nominal factors for nonrectangular regions, related to the non-rectangular structure introduced by optional dimensions."},{"cited_title":"Algorithms for hyper-parameter optimization","cited_arxiv_id":null,"evidence_quote":"Motivates hierarchical input spaces through hyperparameter optimization, one of the application domains the paper targets."}],"review_version":1}