{"id":"d230f1c1-330e-4752-af00-6f8bb44a2489","arxiv_id":"2501.15273","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"Empty-space search with a Lennard-Jones potential, guided by a human-in-the-loop visual interface and a trained neural network, finds configurations that outperform random sampling in several optimization tasks.","lead":"This paper introduces a search algorithm and visual tool, GapMiner, that finds unexplored empty regions in high-dimensional datasets and proposes new configurations from those gaps. It could make it cheaper to discover good parameter settings, novel designs, adversarial examples, and reinforcement-learning policies.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The developed-stage ESA-vs-random superiority claim rests on a DNN surrogate used as both selector and verifier; if the surrogate is optimistically biased in empty regions, the measured advantage is an artifact.","rationale":"The reader's weakest assumption—that the DNN surrogate may be unreliable in the low-density regions where ESA operates—is precisely the load-bearing concern. The paper's strongest quantitative claim (Sec. VIII-C) depends on a closed loop: the same DNN selects ESCs and provides the scores for the dominance-area metric. The paper reports 7.9% average error on in-distribution test data, but ESA deliberately targets empty spaces and Appendix E demonstrates that its outputs move progressively outside the initial data distribution. Thus, in-distribution error is not a valid proxy for surrogate accuracy in the explored regions. A systematic optimistic bias would inflate ESA's reward relative to random sampling, which stays closer to the data manifold. The proposed concrete test—re-evaluating selected configurations with the real simulator—would directly determine whether the measured superiority reflects true performance or model extrapolation error. The reader's CONDITIONAL verdict remains appropriate: the methodology is plausible and the RL case study provides some real-world support, but the headline quantitative claim is not yet verified against ground truth. No verdict change is needed because the existing conditional acceptance already hinges on resolving this question.","tokens_in":25203,"tokens_out":4324,"duration_ms":40907,"concrete_test":"Re-run the Sec. VIII-C developed-stage comparison with a ground-truth check: for each method (ESA, RS, RW), take the Pareto-front configurations selected in a representative subset of the 50 runs, evaluate them with the real simulator used in Sec. VII (or a high-fidelity black box), recompute the dominance area from these true values, and compare methods. Additionally, record the surrogate's signed prediction error on the ESA-selected versus RS-selected candidates; if the mean signed error is significantly positive for ESA candidates, the surrogate loop is the likely cause of the reported advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim that ESA is 'much better than random methods' (Sec. VIII-C) is supported by two experiments in which a well-trained DNN (7.9% average error, measured on in-distribution test data) simultaneously evaluates candidate ESCs and supplies the outcome values used to compute the dominance-area reward. ESA deliberately searches low-density or even extrapolated regions (Appendix E shows ESA results moving progressively farther from the initial distribution), so the surrogate's accuracy in those regions is not guaranteed by its in-distribution error. If the DNN systematically overestimates values in the empty spaces ESA targets, then ESA will appear superior to random sampling simply because it finds points where the model is wrong. The self-referential evaluation loop—selecting with the same model that provides ground truth—makes the reported superiority of ESA (0.450 vs 0.409 for RS and 0.413 for RW in the first experiment; 0.453 vs 0.418 and 0.422 in the second) unverifiable from the paper alone. The RL case study (Sec. IX-C) uses real environment returns and provides genuine but narrow evidence; it does not cover the large-scale developed-stage comparisons that back the abstract's headline claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a visual analytics pipeline, GapMiner, for exploring empty regions in high-dimensional datasets. The central algorithmic contribution is the Empty Space Search Algorithm (ESA), a Lennard-Jones-potential-inspired agent-based search that identifies low-density configurations. The pipeline begins with human-in-the-loop exploration, during which users refine ESA-proposed configurations in GapMiner and verify them against a real (or simulated) black box; the verified configurations augment the dataset and progressively train a DNN that eventually takes over the search and refinement roles. The paper reports a user study (N=10) comparing GapMiner users against ESA-only and random-sampling baselines, two developed-stage comparisons in which ESA is claimed to outperform random sampling and random walk under DNN-guided verification, and three additional case studies: wine dataset augmentation, adversarial images in MNIST latent space, and empty-space policy search in MuJoCo's AntDir. The central claim is that the methodology 'consistently produces substantially superior novel configurations compared to conventional randomization-based methods' (Abstract) and that ESA is 'much better than random methods' (Sec. VIII-C).","tokens_in":25430,"tokens_out":4273,"duration_ms":40701,"significance":"If the central claim were established, the work would be a useful contribution to high-dimensional configuration search and visual analytics: it addresses a real problem (efficient discovery of promising, unverified configurations), and it combines a scalable approximate empty-space search with a human-in-the-loop interface in a way that is more actionable than prior Delaunay-based or flow-line approaches. Genuine strengths include the user study, which uses true verification of the system simulator for the initial-stage comparison and shows a significant advantage for GapMiner users over the baselines; the RL case study, which evaluates discovered policies against actual environment returns; the closed-form derivation connecting PCA updates to original-space coordinate changes (App. A); and the explicit complexity analysis of ESA. However, the quantitative evidence for the developed-stage superiority claim is self-referential: a single DNN surrogate is used both to select candidate ESCs and to compute the outcome values that determine the dominance-area reward. Because ESA deliberately searches low-density or extrapolated regions (App.","major_comments":[{"comment":"The developed-stage comparisons use one well-trained DNN (7.9% average error, measured on in-distribution test data) as both the critic that selects promising ESCs and the black box that verifies their outcome values. Since ESA deliberately searches low-density regions and Appendix E shows that ESA results progressively move away from the initial distribution, the surrogate's accuracy in exactly the regions where ESA operates is not established by its in-distribution error. If the DNN is optimistically biased in those empty regions, the reported dominance-area advantages (0.450 vs. 0.409 for RS and 0.413 for RW; 0.453 vs. 0.418 and 0.422 in the second experiment) could be an artifact of the model's extrapolation behavior rather than a real property of the search. This experiment is the load-bearing evidence for the abstract's claim of 'consistently substantially superior' novel configurations. The authors should validate at least a subset of the developed-stage ESCs against the true simulator (which is already used elsewhere in the paper), or otherwise provide independent evidence of surrogate accuracy in the searched empty regions.","section":"Sec. VIII-C"},{"comment":"The wine case study assigns quality labels to newly generated ESC instances as 'the same as the nearest existing neighbor.' The paper then reports that DNN accuracy improved from 60% to 80% and that numerous innovative wine instances with low free sulfur dioxide were found. But the accuracy improvement is measured against these self-generated nearest-neighbor labels, not against real wine quality, and the paper concedes 'we were unable to verify these results in the field.' This is a circular step: the ESA generates points whose labels are defined by nearest-neighbor propagation, and the DNN is trained and evaluated on those same propagated labels. The wine study should be explicitly framed as a proof-of-concept on synthetic labels; it cannot serve as domain validation of the pipeline's ability to find genuinely better wines.","section":"Sec. IX-A"},{"comment":"The user study's statistical analysis shows that GapMiner users significantly outperform both baselines, but ESA-only is not significantly different from random sampling (p=0.580 for the x2-x3 pair in the Tukey HSD). This is consistent with the paper's framing of the user as essential in the initial stage, but it undercuts any interpretation of the ESA algorithm itself as superior to random methods without the DNN or user in the loop. The abstract's blanket statement that the methodology 'consistently produces substantially superior novel configurations compared to conventional randomization-based methods' therefore requires the developed-stage evidence to carry the weight; as noted above, that evidence is currently circular. The claim should be scoped to the full pipeline and supported by non-circular validation, or the abstract should be revised.","section":"Sec. VIII-A, Table III"}],"minor_comments":[{"comment":"ESA has many free parameters (k, sigma, n, alpha, gamma, delta, j), and only a single parameter setting (App. D) is reported. A sensitivity analysis or at least a brief discussion of parameter choice across the different case studies would greatly improve reproducibility.","section":"Sec. IV-B and App. D"},{"comment":"The baseline protocol is described as 'calculated the Pareto front estimated by the naive neural network, and then verified the naive Pareto front.' It should be clarified whether the same (or an equally trained) DNN was available to the GapMiner users during their search; otherwise the comparison may conflate algorithm performance with the quality of the DNN used for candidate selection.","section":"Sec. VIII-A"},{"comment":"The cos-MDS neighbor visualization is demonstrated on three synthetic shapes, but no quantitative error metric is given for how well the 2D embedding preserves the original high-dimensional cosine distances. Adding a residual or stress measure would strengthen the claim of faithful topology visualization.","section":"Sec. V-E and App. B"},{"comment":"The SUS usability score is based on only 5 of the 10 participants, and the paper acknowledges this is an initial test. Reporting the number of responses alongside the score is appropriate, but the limitation should be restated in the main text rather than only implicitly.","section":"Sec. VIII-B"},{"comment":"In the RL case study, the empty-space policies are derived from the last 20 checkpoints, and the reported 'Empty-Space Policy' appears to be the best of the 20 candidates. Reporting the distribution of returns across the 20 empty-space policies (rather than only max and mean of one policy's episodes) would help assess selection bias and variance.","section":"Sec. IX-C"},{"comment":"There are numerous typographical and formatting artifacts, e.g., 'V oid' in the title, 'T erms' in the index, 'ANVOA' in Appendix D, 'Mujoco' vs. 'MuJoCo', and inconsistent use of 'naive'/'naïve.' These should be cleaned up in a revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the journal's scope (visual analytics, high-dimensional analysis) and contains a genuinely interesting system and several sound components. However, the headline quantitative claim relies on a DNN surrogate used as both selector and verifier in the developed-stage experiments, and the wine study uses propagated nearest-neighbor labels as ground truth. I would not recommend acceptance until the developed-stage ESA-vs-random comparisons are re-run with true simulator returns on at least a meaningful subset (the authors already have access to the CloudPhysics simulator), or until the claims are substantially softened and the circularity explicitly disclosed. The user study and RL case study provide real but narrower evidence; they should be foregrounded, with the surrogate-based experiments re-labeled accordingly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the empty-space search idea is genuinely novel and the visual analytics system looks well built, but the abstract's claim that ESA is 'much better than random methods' rests on a self-referential DNN loop. The developed-stage comparisons in Sec. VIII-C use one well-trained surrogate (7.9% test error) both to select ESCs and to supply the outcome values that define the dominance-area reward. Since ESA deliberately explores low-density and extrapolated regions—Appendix E shows results drifting farther from the initial distribution—the surrogate's in-distribution accuracy does not guarantee its behavior there. If the DNN is optimistically biased in the voids, ESA's apparent advantage (0.450 vs 0.409/0.413) is an artifact. That is a real soft spot, not a nitpick.\n\nWhat the paper does well: the Lennard-Jones based agent search is a clever, parallelizable alternative to Delaunay triangulation, and I don't know prior work using it for empty-space configuration discovery. GapMiner is thoughtfully designed, with design goals tied to interface components, and the cos-MDS neighbor visualization is a nice contribution. The user study (N=10) shows GapMiner users beat ESA-only and random under true verification, and the paper honestly reports that ESA alone is not significantly better than random (p=0.580). That is credible evidence for the HITL pipeline, if not for ESA's standalone power. The RL case study uses real environment returns and finds empty-space policies with higher returns than training policies—genuine, though narrow, evidence.\n\nOther soft spots: the wine augmentation labels generated ESCs by nearest-neighbor and then reports DNN accuracy improving from 60% to 80%; that is a less meaningful result because the labels come from the existing data. Several case studies (MNIST adversarial images, CheetahDir) lack baselines. The code/data are not released, which makes the surrogate experiments unverifiable. The paper does acknowledge some limitations in the discussion (independence assumption, random initialization), but it does not flag the circular evaluation.\n\nBottom line: this deserves a serious referee. The idea is worth engaging, but the headline claim needs either true-outcome verification in the developed stages or a much more hedged statement. I'd send it to review with a request for major revision.","headline":"Novel L-J-based empty-space search with a solid user study, but the paper's strongest quantitative claim is undermined by a surrogate-as-truth evaluation loop.","tokens_in":26002,"tokens_out":2664,"would_cite":true,"duration_ms":23958,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Empty regions of high-dimensional data are reservoirs of novel configurations that a physics-inspired search can find far more reliably than random sampling.","keywords":["empty space search","high-dimensional data","Lennard-Jones potential","visual analytics","human-in-the-loop machine learning","configuration optimization","adversarial learning","reinforcement learning"],"falsifier":"Run the developed-stage comparison with the true objective instead of the DNN surrogate: take the 1,000- and 3,000-configuration cache datasets, have ESA, random sampling, and random walk each propose 1,500 configurations, then verify the top candidates on the real simulator or trace and compare Pareto dominance areas. If random sampling matches or beats ESA under real verification, the claimed advantage is an artifact of the surrogate's low-density extrapolation.","tokens_in":24942,"feed_emoji":"🕳️","tokens_out":3987,"duration_ms":33828,"temperature":0.7,"pith_summary":"The paper's central assertion is that the holes in a high-dimensional dataset—regions where no verified configuration exists—are not dead space but reservoirs of potentially valuable configurations. It proposes an Empty Space Search Algorithm (ESA) that treats data points as particles and sends repelled agents into these gaps, then wraps the search in a visual-analytics system (GapMiner) and a human-in-the-loop to AI pipeline that trains a neural network to eventually run the search alone. The authors claim this methodology consistently yields substantially better novel configurations than random sampling, and demonstrate it on cache configuration, wine chemistry, adversarial MNIST images, and reinforcement-learning policies. If true, expensive evaluation budgets in high-dimensional design spaces could be spent on a small number of high-value probes instead of large random sweeps.","feed_headline":"Empty-space search finds better configurations than random sampling","feed_subtitle":"A physics-inspired agent search targets the gaps in high-dimensional data, claiming superior novel designs and policies across four case…","key_machinery":"The load-bearing object is the Empty Space Search Algorithm (ESA), driven by the Lennard-Jones potential $V(r)=4\\epsilon\\left[(\\sigma/r)^{12}-(\\sigma/r)^6\\right]$. Each agent sums repulsive and attractive forces from its $k$ nearest data points, moves along the resultant force direction, and returns its trajectory as raw ESCs; a momentum term broadens the search once a DNN can score candidates, and gradient ascent on the DNN refines the final pick. The agent formulation is what keeps the search at $O(dknp + np\\log N)$ instead of Delaunay's $O(N^{\\lceil d/2\\rceil})$, and it is what makes the method parallelizable across many agents.","core_discovery":"The central discovery is that 'empty space configurations' (ESCs)—points in the data space far from all known verified points—are disproportionately likely to be novel, high-performing, or adversarial, and that a physics-inspired agent search can find them without constructing the Delaunay triangulation whose complexity explodes in high dimensions. ESA places agents in the space, lets Lennard-Jones repulsion and attraction drive them into equilibrium voids, records the trajectory as candidate ESCs, and later uses a trained DNN plus gradient ascent to refine the best candidates. Across a user study and three comparison experiments, the paper reports that ESA significantly outperforms random sampling and random walk in Pareto dominance area, and in the developed stages with a DNN surrogate ESA is much better than random methods.","pith_inferences":["A natural testable extension is to replace the DNN surrogate with the real objective in the developed-stage comparisons; if ESA still beats random sampling, the advantage is a property of the search geometry rather than of surrogate extrapolation.","Because ESA's stopping criterion is force magnitude below a threshold, the method effectively defines 'empty' relative to the current dataset; active learning that selects ESCs minimizing DNN uncertainty could make the training loop converge faster.","The same empty-space machinery could serve as a data-augmentation generator for imbalanced classification: the wine study already improves quality-prediction accuracy from 60% to 80%, and a similar strategy could probe out-of-distribution examples for robustness testing.","The reinforcement-learning result hints at a broader principle: interpolating between nearby high-performing checkpoints in weight space may land on superior policies, a claim that could be checked across other MuJoCo environments."],"forward_implications":["If ESA reliably finds high-value ESCs, design spaces such as storage-cache configurations can be explored with far fewer expensive verifications than random or grid search requires.","The DNN-as-critic pipeline suggests that once a surrogate model is accurate enough, the human-analyst role can shift to mentoring rather than direct search, and the same workflow can autocomplete finding adversarial examples in latent spaces.","In reinforcement-learning policy-parameter spaces, empty-space search can yield policies whose returns exceed the best training checkpoints, pointing to a less computation-intensive way to improve trained agents.","The Pareto dominance-area gains reported in the systems case (from 0.27 to 0.56) imply that the approach can expand the known trade-off frontier rather than merely refine existing designs."],"supporting_citations":[{"why":"Invokes the curse of dimensionality to motivate why empty-space search in high dimensions is hard.","marker":"[5]"},{"why":"Supplies the Lennard-Jones potential that drives the ESA agents' repulsion and attraction.","marker":"[35]"},{"why":"Represents the prior empty-space identification work in 3D that this paper extends to high dimensions.","marker":"[55]"},{"why":"Defines the Pareto dominance area metric used to measure ESA's superiority over random baselines.","marker":"[10]"},{"why":"Provides the CloudPhysics trace and simulator environment used in the systems application example.","marker":"[60]"},{"why":"Gives the red wine dataset used for the domain-agnostic case study.","marker":"[52]"},{"why":"Supplies the Soft Actor-Critic algorithm whose policy checkpoints form the search space in the RL case study.","marker":"[25]"},{"why":"Provides the MuJoCo simulation environment used to evaluate empty-space policies.","marker":"[57]"}],"fun_headline_variants":["GapMiner maps data voids to find novel configurations","Physics-inspired agents beat random search in data gaps","Empty-space search uncovers high-value configurations","Agent search in voids outperforms random sampling","Data voids hold better designs, new algorithm finds them"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The developed-stage quantitative evidence assumes the trained DNN (with 7.9% average error on in-distribution data) is trustworthy both for selecting ESCs and for verifying their value; if the surrogate errs systematically in the low-density regions where ESA operates, the measured superiority over random sampling could be an artifact of that extrapolation rather than a real property of the search.","fun_headline_variants_meta":{"raw":{"variants":["GapMiner maps data voids to find novel configurations","Physics-inspired agents beat random search in data gaps","Empty-space search uncovers high-value configurations","Agent search in voids outperforms random sampling","Data voids hold better designs, new algorithm finds them"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000194,"raw_usage":{"total_tokens":1353,"prompt_tokens":944,"completion_tokens":409,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":560,"completion_tokens_details":{"reasoning_tokens":337}},"tokens_in":560,"tokens_out":409,"duration_ms":4156,"temperature":1.0,"reasoning_tokens":337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:26:27.092503+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the developed-stage comparison with the true objective instead of the DNN surrogate: take the 1,000- and 3,000-configuration cache datasets, have ESA, random sampling, and random walk each propose 1,500 configurations, then verify the top candidates on the real simulator or trace and compare Pareto dominance areas. If random sampling matches or beats ESA under real verification, the claimed advantage is an artifact of the surrogate's low-density extrapolation.","supporting_citations":[{"cited_title":"Dynamic Programming","cited_arxiv_id":null,"evidence_quote":"Invokes the curse of dimensionality to motivate why empty-space search in high dimensions is hard."},{"cited_title":"On the Determination of Molecular Fields.–—I. From the V ariation of the Viscosity of a Gas with Temperature","cited_arxiv_id":null,"evidence_quote":"Supplies the Lennard-Jones potential that drives the ESA agents' repulsion and attraction."},{"cited_title":"Real-Time Visualization of Protein Empty Space with V arying Parameters","cited_arxiv_id":null,"evidence_quote":"Represents the prior empty-space identification work in 3D that this paper extends to high dimensions."},{"cited_title":"On Using the Hypervolume Indicator to Compare Pareto Fronts: Applications to Multi-criteria Optimal Experimental Design","cited_arxiv_id":null,"evidence_quote":"Defines the Pareto dominance area metric used to measure ESA's superiority over random baselines."},{"cited_title":"Efficient MRC Construction with SHARDS","cited_arxiv_id":null,"evidence_quote":"Provides the CloudPhysics trace and simulator environment used in the systems application example."},{"cited_title":"https://www.kaggle.com/ datasets/midouazerty/redwine","cited_arxiv_id":null,"evidence_quote":"Gives the red wine dataset used for the domain-agnostic case study."},{"cited_title":"Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 13 Stochastic Actor","cited_arxiv_id":null,"evidence_quote":"Supplies the Soft Actor-Critic algorithm whose policy checkpoints form the search space in the RL case study."},{"cited_title":"MuJoCo: A physics engine for model-based control","cited_arxiv_id":null,"evidence_quote":"Provides the MuJoCo simulation environment used to evaluate empty-space policies."}],"review_version":1}