REVIEW 4 major objections 7 minor 13 references
instancespace: a Python Package for Insightful Algorithm Testing through Instance Space Analysis
T0 review · 4 major / 7 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read A Python package implements the full six-stage Instance Space Analysis pipeline.
desk verdict A useful Python port of an established methodology whose central replication claim is currently unverified, and whose example code is sloppy. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the InstanceSpace class, which orchestrates a pipeline of stage classes—PreprocessingStage, PrelimStage, SiftedStage, PilotStage, PythiaStage, CloisterStage, and TraceStage—connected through a StageBuilder that resolves data dependencies. Each stage implements a standard interface with static typing, and a JSON options file configures the parameters for every step. The central transformation is the mapping from high-dimensional instance-feature data with algorithm-performance measurements into a two-dimensional instance space; once built, that space is what makes the algorithmic footprints and SVM-based algorithm recommendations possible.
What would settle it
Run the package and the original implementation on the same metadata and options file, then compare the resulting 2D coordinates, PYTHIA's algorithm-selection accuracy, and TRACE's footprint boundaries; any material divergence on a standard dataset would refute the claimed replication.
Extended reading notes
Core claim
The paper's central claim is that the complete ISA pipeline can be captured in a modular Python package without losing the functionality of the original reference software. The pipeline is described as six stages: Preprocessing cleans and normalises metadata; PRELIM sets a performance threshold and binarises good versus bad performance; SIFTED selects the features most correlated with algorithm performance; PILOT projects the high-dimensional feature space to two dimensions, either analytically via eigenvalue projection or numerically via BFGS minimisation; CLOISTER bounds the space by projecting feature limits; PYTHIA trains Support Vector Machines to recommend algorithms for untested instances; and TRACE constructs geometric footprints with area, density, and purity metrics. The authors report that these stages run in a dependency-resolved pipeline orchestrated by an InstanceSpace class, and that the package produces instance-space plots and footprint summaries like those of the original implementation.
Load-bearing premise
The load-bearing premise is that each reimplemented stage faithfully reproduces the behaviour of the original reference software, since the paper reports no numerical benchmark, regression test, or direct comparison against that software.
Editorial extensions
If this is right
- If the central claim holds, researchers can run ISA locally in Python without the original MATLAB environment.
- The stage-based pipeline with dependency resolution lets users execute the full analysis or run individual stages independently, so only the needed step can be recomputed.
- PYTHIA's SVM models are claimed to recommend the best algorithm for new, untested instances, reducing the need to run every benchmark exhaustively.
- TRACE's footprint metrics—area, density, purity—give a quantitative summary of where each algorithm is strong and where it is weak.
- CLOISTER's boundary projection and synthetic-instance generation are aimed at uncovering gaps in existing test suites and filling them.
Reading between the lines
- A direct numerical comparison of the Python stages' projections and footprints against the original implementation on identical metadata would be a natural validation step and would make the replication claim independently checkable.
- Because PYTHIA is a separate stage, the modular design suggests the SVM core could be swapped for other classifiers or regressors without rewriting the pipeline, a testable extension of the claimed architecture.
- The staging API also implies the methodology could be carried into new problem domains—such as time-series or safety-critical scenario testing—as long as instance features and performance metadata are supplied.
- If the replication is faithful, the package's Python integration may lower the barrier to making ISA a routine part of benchmark-driven AI validation rather than a specialist analysis step.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces `instancespace`, a Python package that implements the six-stage Instance Space Analysis (ISA) pipeline of Smith-Miles and Muñoz [2]: PRELIM (metadata preparation), SIFTED (feature selection), PILOT (dimensionality reduction), CLOISTER (instance-space boundaries), PYTHIA (SVM-based algorithm selection), and TRACE (footprint construction). The manuscript describes the motivation for ISA, the package architecture (modular stages orchestrated by an `InstanceSpace` class and `StageBuilder`), illustrative code for loading metadata/options and running the pipeline, example footprint and SVM output tables, and expected impact for algorithm testing and deployment. It also provides code metadata (Table 1) and a link to a repository.
Significance. If the replication claim in Section 1 is substantiated, `instancespace` would be a valuable open-source Python implementation of ISA, lowering the barrier to adoption, enabling integration with Python-based benchmarking tools such as IOHexperimenter, and potentially improving the scalability and deployability of ISA workflows. The paper's contribution, however, rests entirely on the claim that the package replicates the established MATLAB ISA toolkit [12]. That claim is not supported by any comparative evaluation, numerical regression test, or benchmark in the manuscript, and the illustrative code appears inconsistent with the released package. Consequently, the significance is conditional on the authors providing evidence that the six stages reproduce the reference behavior.
major comments (4)
- [Section 1, first paragraph and Section 6] The central claim that `instancespace` "replicates the functionality of the MATLAB code as described in [2]" is not supported by any evidence reported in the manuscript. No comparison against the MATLAB ISA toolkit [12] is presented: no feature-selection lists, no 2D instance-space projections, no footprint area/density/purity values, and no SVM accuracy metrics are compared on the same input metadata. To establish the replication claim, the authors should add a validation section that runs both implementations on a common benchmark dataset and shows quantitative agreement (or documents and explains any differences). Without this, the paper's central contribution is unverified.
- [Section 4.1 and 4.2] The illustrative code imports from `matilda` (`from matilda import instancespace`, `from matilda.stages.pilot import PilotStage`) while the package, repository, and article title are all named `instancespace` (with repository `pyInstanceSpace`). This inconsistency strongly suggests the code examples were not executed against the released artifact, undermining the reproducibility of the paper's own example. All code snippets must be updated to import from the actual package name and the API must be verified against the released version.
- [Section 4.2, bulleted list] The stage descriptions in the bulleted list contradict the stage definitions in Section 2. `PilotStage` is described as "Tests algorithms on selected instances" (but PILOT performs dimensionality reduction), and `CloisterStage` is described as "Groups similar instances based on their performance" (but CLOISTER defines the boundaries of the instance space). These misdescriptions should be corrected to match the methodological descriptions in Section 2 and the original ISA methodology.
- [Section 5, 'Impact'] The paper claims that Python's speed and scalability make `instancespace` ideal for large-scale data and frequent analysis, and that MATILDA "suffers from scalability issues due to its MATLAB-native computing code." No timing, memory, or scalability measurements are provided for either system, and no threshold for "large-scale" is given. Either add benchmark evidence supporting these assertions or remove/qualify them as unsubstantiated.
minor comments (7)
- [Section 3.1] The term "beta performance" appears in the description of Algorithm Footprint Analysis but is never defined; clarify whether this refers to a specific footprint type in TRACE or the beta parameter used elsewhere in ISA.
- [Section 4.1] The line `script_dir = Path(os.path.abspath(''))` contains two single quotes, which is a typo; this should be a proper path expression (for example, using `__file__` or an explicit relative path) so the example is runnable as shown.
- [Section 4.2] The import `from matilda.stages. preprocessing import PreprocessingStage` contains an extra space before `preprocessing`; this typo should be fixed in the code listing.
- [Section 4.4] The printed `data_matrix` appears to have more than two columns, yet the text describes the instance space as a 2D projection; clarify whether this matrix is the full transformed feature matrix or a reduced representation, and show actual output that matches the dimensionality claimed.
- [Table 2 and Table 3] The table headings (e.g., "Area Good Normalized", "Density Good Normalized", "Purity Good") are not defined in the text; provide definitions of the normalization and the footprint metrics so the reported values are interpretable.
- [Section 4.5] The text states that footprints are clustered using DBSCAN, but Section 2's TRACE description does not mention clustering or DBSCAN; specify the clustering algorithm and its parameters, and relate them to the original ISA implementation.
- [References [13]] The statement that MATILDA suffers from scalability issues is presented without a citation or evidence; either supply a reference or soften the claim.
Circularity Check
No circularity: the paper's central claim is an external replication claim about a software port, not a derivation that reduces to its own inputs.
full rationale
This is an implementation paper, not a derivation or prediction study. The central claim in Section 1 is that instancespace 'replicates the functionality of the MATLAB code as described in [2]'. That is an external equivalence claim: the reference standard is the prior MATLAB toolkit [12] and the ISA methodology [2]. The package's behavior is meant to mirror that external reference, so there is no fitted parameter renamed as a prediction, no self-definitional equation, and no uniqueness theorem imported from the authors. The presence of self-citations (e.g., [2], [10], [11], [12] include the corresponding author) is not load-bearing circularity because the cited prior work is the methodology and original code being reimplemented, not an unverified premise that makes the package correct by construction. The paper does have validation gaps: it provides no numerical comparison, regression test, or benchmark against the MATLAB toolkit to support the replication claim, and the illustrative code in Section 4 imports from 'matilda' rather than the released 'instancespace' package. These are reproducibility and support weaknesses, not circularity. Therefore no circular step can be exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
assumptions (2)
- domain assumption Instance Space Analysis methodology as defined by Smith-Miles and Munoz [2] is the correct target behavior for the package.
- domain assumption The original MATLAB ISA code [12] is a faithful implementation of [2] and serves as the behavioral reference.
Cite this review
Pith. "Pith review of instancespace: a Python Package for Insightful Algorithm Testing through Instance Space Analysis." pith.science (2026). https://pith.science/paper/FOY7FSI7
@misc{pith2026250116646,
author = {Pith},
title = {Pith review of: instancespace: a Python Package for Insightful Algorithm Testing through Instance Space Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/FOY7FSI7}},
note = {Machine review of arXiv:2501.16646}
}
read the original abstract
Instance Space Analysis is a methodology to evaluate algorithm performance across diverse problem fields. Through visualisation and exploratory data analysis techniques, Instance Space Analysis offers objective, data-driven insights into the diversity of test instances, algorithm behaviour, and algorithm strengths and weaknesses. As such, it supports automated algorithm selection and synthetic test instance generation, increasing testing reliability in optimisation, machine learning, and scheduling fields. This paper introduces instancespace, a Python package that implements an automated pipeline for Instance Space Analysis. This package supports research by streamlining the testing process, providing unbiased metrics, and facilitating more informed algorithmic design and deployment decisions, particularly for complex and safety-critical systems.
Figures
Reference graph
Works this paper leans on
-
[2]
K. Smith-Miles, M. A. Muñoz, Instance space analysis for algorithm testing: Methodology and software tools, ACM Comput. Surv. 55 (12) (mar 2023). doi:10.1145/3572895. URL https://doi.org/10.1145/3572895
-
[12]
M. Muñoz, K. Smith-Miles, Instance space analysis: A toolkit for the assessment of algorithmic power (2020). doi:10.5281/zenodo.4750845. URL https://github.com/andremun/InstanceSpace
-
[1]
Hooker, Testing heuristics: We have it all wrong, J
J. Hooker, Testing heuristics: We have it all wrong, J. Heuristics 1 (1) (1995) 33–42. doi:10.1007/BF02430364
-
[3]
N. Neelofar, A. Aleti, Identifying and explaining safety-critical scenarios for au- tonomous vehicles via key features, ACM Trans. Softw. Eng. Methodol. 33 (4) (Apr. 2024). doi:10.1145/3640335. URL https://doi.org/10.1145/3640335
doi:10.1145/3640335 2024
-
[4]
V . Crespo-Rodriguez, N. Neelofar, A. Aleti, B. Turhan, Instance space analysis of testing of autonomous vehicles in critical scenarios, ACM Trans. Softw. Eng. Methodol.Just Accepted (Oct. 2024). doi:10.1145/3699596. URL https://doi.org/10.1145/3699596
-
[6]
M. E. Scherer, R. R. Hill, B. J. Lunday, B. A. Cox, E. D. White, Applying in- stance space analysis for metaheuristic selection to the 0–1 multidemand multi- dimensional knapsack problem, Computers & Operations Research 170 (2024) 106747. doi:https://doi.org/10.1016/j.cor.2024.106747. URL https://www.sciencedirect.com/science/article/pii/ S0305054824002193
-
[7]
D. Van Bulck, D. Goossens, J.-P. Clarner, A. Dimitsas, G. H. Fonseca, C. Lamas- Fernandez, M. M. Lester, J. Pedersen, A. E. Phillips, R. M. Rosati, Which algo- rithm to select in sports timetabling?, European Journal of Operational Research 318 (2) (2024) 575–591. doi:https://doi.org/10.1016/j.ejor. 12 2024.06.005. URL https://www.sciencedirect.com/scienc...
doi:10.1016/j.ejor 2024
-
[8]
N. Andrés-Thió, M. A. Muñoz, K. Smith-Miles, Characterising harmful data sources when constructing multi-fidelity surrogate models, Artificial Intelligence 336 (2024) 104207. doi:https://doi.org/10.1016/j.artint. 2024.104207. URL https://www.sciencedirect.com/science/article/pii/ S0004370224001437
Show all 13 references
-
[9]
Skålnes, M
J. Skålnes, M. Ben Ahmed, L. M. Hvattum, M. Stålhane, New benchmark in- stances for the inventory routing problem, European Journal of Operational Re- search 313 (3) (2024) 992–1014. doi:https://doi.org/10.1016/j. ejor.2023.08.010. URL https://www.sciencedirect.com/science/art...
2024 doi
-
[10]
M. A. Muñoz, K. Smith-Miles, Generating new space-filling test instances for continuous black-box optimization, Evol. Comput. 28 (3) (2020) 379–404.doi: 10.1162/evco_a_00262. URL https://doi.org/10.1162/evco_a_00262
2020 doi
-
[11]
E. Yap, M. A. Muñoz, K. Smith-Miles, Informing multiobjective optimization benchmark construction through instance space analysis, IEEE Transactions on Evolutionary Computation 26 (6) (2022) 1246–1260. doi:10.1109/TEVC. 2022.3205165
2022
-
[13]
Smith-Miles, M
K. Smith-Miles, M. Muñoz, Neelofar, Melbourne algorithm test instance li- brary with data analytics (MATILDA), Available at https://matilda. unimelb.edu.au (2020)
2020
-
[14]
de Nobel, F
J. de Nobel, F. Ye, D. Vermetten, H. Wang, C. Doerr, T. Bäck, Iohexperi- menter: Benchmarking platform for iterative optimization heuristics, arXiv e- prints:2111.04077 (nov 2021). arXiv:2111.04077. URL https://arxiv.org/abs/2111.04077 13
2021 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.