{"id":"2fed728a-e25f-4c78-9f97-e110ca0e774e","arxiv_id":"2508.20254","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Combining outlier-based novelty scores with strategic non-smooth exploration makes autonomous experiments sample more diverse data and predict a known physical property better than standard optimization.","lead":"An autonomous experiment framework scores each measurement by how unusual it is and then strategically samples under-explored regions, instead of only optimizing a known target. On a known ferroelectric dataset and a real microscope, this approach measures a more diverse set of responses than standard Bayesian optimization while still improving prediction of a known property.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The diversity metric in Fig. 4b is the same quantity the novelty scores are designed to maximize, so the 'diversity of explored phenomena' result is partly tautological and is never validated against the known ground-truth domain classes.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the primary diversity metric overlaps with the optimized objective and is not validated against physical ground truth. My reading of the manuscript confirms this. The quantitative assessment uses variability of measured hysteresis loops as the discovery metric, but the novelty scores are themselves measures of loop unusualness, and the acquisition function drives the experiment toward high-novelty locations. Thus Fig. 4b largely demonstrates that the acquisition function does what it was designed to do, not that new physical phenomena were discovered. The NME improvement in Fig. 4a is a meaningful independent result, but it is based on single runs, lacks error bars, and concerns prediction of a known scalarizer rather than discovery of unknown physics. I do not see an internal inconsistency or a fraud concern; the method is plausible and the code is promised. But the abstract's claim about 'discovering previously unobserved phenomena' is not currently supported by the evidence. This matches the reader's CONDITIONAL verdict, so no verdict change is needed; the paper should be revised to ground the diversity claim in physical classes or explicitly soften the discovery language.","tokens_in":9809,"tokens_out":2971,"duration_ms":33083,"concrete_test":"Using the pre-acquired BEPS dataset and its annotated ground-truth domain structure (Supp. Fig. 1), label every pixel by physical class (out-of-plane up/down, in-plane, domain wall). Re-run or replay the four AE variants (scalarizer, IF-novelty, NN-novelty, INS2ANE variants) and compute the class-coverage entropy and the fraction of measured points in the minority domain-wall class over the 200 measurements. If the novelty-driven and INS2ANE methods do not achieve materially higher class coverage or minority-class sampling than the scalarizer-driven AE or random sampling, the diversity-of-phenomena claim is not supported. Additionally, compute the Spearman correlation between each measured loop's IF/NN novelty score and its contribution to the Fig. 4b variability metric; a high correlation confirms the metric is redundant with the optimized objective.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that INS2ANE 'significantly increases the diversity of explored phenomena' rests on the variability metric defined in the Quantitative Assessment section: the mean, across measurement timesteps, of the standard deviation of the measured hysteresis loops. But the novelty scores being optimized (NN, IF, etc.) are explicitly computed from how unlike the other measured loops a loop is, and the acquisition function selects locations predicted to have high novelty. Higher measured-loop variability is therefore a near-direct consequence of optimizing a novelty score, not an independent measurement of physical discovery. The paper's own ground-truth dataset contains known physical classes (out-of-plane domains, in-plane domains, domain walls), yet the authors never report whether novelty-driven AEs sample a broader set of these classes than the scalarizer-driven AE or random sampling. The independent NME improvement in Fig. 4a is a real, non-tautological result, but it is presented from single runs without error bars and it measures surrogate accuracy for a pre-defined scalarizer, not the discovery of unobserved phenomena. Consequently, the evidence supports 'novelty scoring changes where the AE measures and can improve surrogate accuracy,' but not the stronger claim about discovering previously unobserved physical phenomena.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes INS2ANE (Integrated Novelty Score-Strategic Autonomous Non-Smooth Exploration), a framework for autonomous experiments that augments Bayesian optimization with a novelty scoring module and a strategic sampling module (SANE). The goal is to move beyond optimizing a predefined scalar descriptor and instead promote the discovery of unexpected or unknown physical phenomena. The authors evaluate five novelty scoring methods (Distance to Centroid, Nearest Neighbors, IsolationForest, OneClassSVM, LocalOutlierFactor) on a pre-acquired band excitation piezoresponse spectroscopy (BEPS) dataset of a ferroelectric PbTiO3 sample with known ground-truth domain structures. After selecting NN and IF as suitable, they compare a scalarizer-driven AE, novelty-driven AEs, and INS2ANE variants using two metrics: normalized mean error (NME) of the predicted physical scalarizer and the variability of measured hysteresis loops. They also implement the approach in real autonomous scanning probe microscopy experiments, reporting four realizations. The paper claims that INS2ANE significantly increases the diversity of explored phenomena and enhances the likelihood of discovering previously unobserved phenomena.","tokens_in":10005,"tokens_out":3033,"duration_ms":29584,"significance":"If the central claim were rigorously supported, this work would be a useful contribution to the growing literature on autonomous experimentation by directly addressing the limitation that most AEs only optimize predefined targets. The strength of the paper is that it demonstrates a concrete implementation on both a model dataset and a real experimental platform, and makes code available on GitHub. The exploration of five different novelty scoring methods is thorough. However, the quantitative evidence does not yet support the strong claim of 'increased diversity of explored phenomena' because the primary diversity metric is conceptually tied to the objective being optimized, and the comparisons rest on single runs without error bars. The independent NME result is real but measures surrogate accuracy for a pre-defined scalarizer, not the discovery of new physics. The paper is therefore of interest but needs substantial revision to support its headline claim.","major_comments":[{"comment":"The variability metric used to demonstrate 'increased diversity' is the mean standard deviation among measured hysteresis loops; this is essentially the same quantity the novelty scores (NN, IF) are designed to maximize, since both assess how different a measured loop is from the other measured loops. The diversity result in Fig. 4b is therefore partly tautological and does not independently establish that the AE discovers previously unobserved physical phenomena. Since the dataset has known ground-truth classes (out-of-plane domains, in-plane domains, domain walls), I recommend reporting the coverage of these classes over the course of each AE as a non-tautological test, e.g., the fraction of measurement locations that fall into each class, or a class-entropy metric.","section":"Quantitative Assessment, Fig. 4b"},{"comment":"The comparison of NME across AE approaches is based on a single run per configuration, with no error bars or multiple seeds. The initial random points and the stochastic acquisition process make these curves noisy; for instance, the scalarizer-driven AE shows a non-monotonic behavior that is hard to interpret from one trajectory. Without multiple realizations and a statistical summary (mean/median with dispersion), the claim that novelty-driven methods 'end with significantly lower NMEs' is not quantitatively supported. Please provide at least 5–10 independent runs per configuration and report distribution statistics or confidence intervals.","section":"Quantitative Assessment, Fig. 4a"},{"comment":"The selection of NN and IF as the novelty scoring methods is made after inspecting all five scores on the full model dataset, using domain knowledge about the in-plane domains' low SNR and the desired behavior. This post hoc selection, performed on the same data later used for evaluation, risks inflating the reported performance of the chosen methods. The paper would be stronger if it either (a) reports results for all five novelty scorers, (b) uses a held-out subset for method selection, or (c) clearly frames the study as a demonstration of a workflow rather than a head-to-head benchmark of the selected methods.","section":"Implementation with a Pre-Acquired Model Dataset"},{"comment":"The NME metric evaluates how accurately each AE predicts the pre-defined scalarizer (loop area), which is an abstraction of 'physical understanding' but not a direct measure of discovering previously unobserved phenomena. The claim that novelty-driven approaches 'enhance model prediction accuracy' is supported, but the paper's broader claim about enhancing the likelihood of discovering previously unobserved physical phenomena would require a direct evaluation against the known physical classes in the ground-truth dataset. Concretely, does the scalarizer-driven AE fail to sample certain classes that the novelty-driven AEs do sample? This can be quantified with the same ground-truth labels already available.","section":"Quantitative Assessment, Fig. 4a and Conclusions"},{"comment":"The real-experiment comparison uses one IF-based realization and three NN-based realizations, with no error bars or statistical test on the variability curves in Fig. 5d. The interpretation of the 'second jump in variability around step-30' as evidence of strategic sampling discovering a novel region is speculative; the same pattern could arise from random fluctuations or differences in sample regions. Please provide more replicates (at least 5–10 per configuration) and quantitative uncertainty estimates, or temper the interpretation accordingly.","section":"Implementation in Autonomous Microscopy, Fig. 5d"}],"minor_comments":[{"comment":"In the comparison to random sampling, the text states 'reaches an average variability of 1.404 (standard deviation: 0.003)'; this appears to be a typo for 0.1404, since the variability values throughout the paper are on the order of 0.1. Please correct.","section":"Quantitative Assessment"},{"comment":"The text says 'While LoC could be useful...' in reference to the LocalOutlierFactor method; this should read 'LOF' for consistency with the earlier enumeration.","section":"Novelty Scoring and Strategic Sampling"},{"comment":"The five novelty-score maps in Fig. 2 are described qualitatively but the color scale is said to be 'arbitrary'; state explicitly whether the scores are normalized to a common range for the figure, and whether the qualitative comparisons (e.g., 'higher novelty') refer to the raw scores.","section":"Figure 2 and surrounding text"},{"comment":"The term 'image-spectral pairs' in the abstract is slightly awkward; consider 'image–spectroscopy pairs' or 'image and spectroscopy data pairs' for clarity.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The central idea is appealing and timely, but the headline claim rests on a metric that is circular with respect to the novelty objective, and the quantitative evaluation lacks error bars. The independent NME result is a positive point, but it measures surrogate accuracy rather than discovery of new phenomena. The paper would be substantially stronger if the authors added a ground-truth class-coverage analysis and repeated runs. The post hoc selection of novelty scorers is also a concern that should be disclosed more clearly or methodologically addressed. I do not see a fundamental flaw in the framework itself, so a major revision is appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper combines two published ideas—outlier-based novelty scores (nearest-neighbor, isolation forest) and the SANE strategic sampling method—into a single loop for autonomous scanning probe microscopy. The integration is straightforward, but the real experiment on a physical microscope is a step beyond simulation, and the authors ship code and data. That is worth something.\n\nThe strongest non-tautological result is Fig. 4a: novelty-driven AEs, with or without INS2ANE, end with lower normalized mean error for the scalarizer prediction (0.15–0.16 vs. 0.30 for the conventional scalarizer-driven AE). That is an independent measure of surrogate accuracy, and it supports the argument that broader sampling improves physical understanding.\n\nThe soft spots are mostly in the diversity claim. The primary metric in Fig. 4b is the mean standard deviation of measured hysteresis loops. The novelty scores are designed to flag loops that differ from the rest, so higher variability is nearly a direct consequence of the acquisition rule. The paper never reports whether novelty-driven AEs actually visit a broader set of the known ground-truth classes (out-of-plane, in-plane, domain walls) relative to the conventional AE. That would have broken the circularity. Also, the choice to keep only NN and IF is made after inspecting the five score maps on the full dataset—post hoc selection that should be flagged. The model-dataset comparisons are single runs, with no error bars in Fig. 4. The real-microscope experiment uses four realizations, but only one IF version, so statistics are thin.\n\nThe abstract's phrase 'significantly increases the diversity of explored phenomena... enhancing the likelihood of discovering previously unobserved phenomena' is stronger than the evidence supports. What is shown is that novelty scoring changes where the AE measures and can improve surrogate accuracy. That is a legitimate contribution, just not the full discovery claim.\n\nI'd send this to a referee, but with a request that the authors either validate the diversity metric against known classes or soften the claim. The paper is useful for people building self-driving labs, and the integration is honest enough to build on. Not a game changer, but a reasonable methods paper after revision.","headline":"A useful integration of novelty scoring and strategic sampling for self-driving microscopy, but the diversity claim rests on a partly circular metric and single-run comparisons; the improved surrogate accuracy is the real result.","tokens_in":10579,"tokens_out":2247,"would_cite":false,"duration_ms":20131,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an autonomous experiment which scores candidate measurements by novelty and deliberately samples under-explored regions discovers more diverse physics than one that optimizes a known physical descriptor, with lower…","keywords":["autonomous experiments","novelty discovery","Bayesian optimization","strategic sampling","scanning probe microscopy","active learning","deep kernel learning","ferroelectric materials"],"falsifier":"One concrete test: on a sample with phases mapped by an independent structural method, run the three acquisition strategies and count how many previously unseen phase regions each one samples; the central claim fails if the novelty-driven and strategic runs do not find more such regions, or if their extra variability is dominated by noise rather than by distinct physical states.","tokens_in":9587,"feed_emoji":"🔬","tokens_out":8415,"duration_ms":78898,"temperature":0.7,"pith_summary":"Autonomous experiments usually aim at a pre-defined target, like maximizing a physical descriptor, and this paper asks whether the same loop can instead be aimed at the unexpected. It introduces INS2ANE, a Bayesian-optimization framework in which the acquisition function uses a novelty score—how unlike previously collected polarization-switching curves a new measurement would be—plus a strategic sampler that pushes measurements into under-sampled regions. On a model ferroelectric dataset, the novelty-driven experiments end with lower prediction error (0.15–0.16 versus 0.30 for the optimizer) and measure more variable, hence more diverse, hysteresis curves than random sampling. The same workflow is demonstrated live on a scanning probe microscope. If the measured-loop variability is accepted as a proxy for discovering new phenomena, the results show that novelty scoring and strategic sampling each contribute to exploration, with the two mechanisms dominant at different stages.","feed_headline":"Novelty-seeking labs beat optimizers at finding diverse physics","feed_subtitle":"On ferroelectric data, novelty-guided runs explore more varied hysteresis loops and beat optimizer error.","key_machinery":"The load-bearing mechanism is the replacement of a physical scalarizer in the acquisition function by a novelty score. Five scores are examined—Distance to Centroid, Nearest Neighbors, Isolation Forest, One-Class SVM, and Local Outlier Factor—each assigning a uniqueness value to a measured hysteresis loop relative to previously collected loops. INS2ANE couples one of these scores (the paper selects Nearest Neighbors or Isolation Forest for its ferroelectric dataset, after rejecting global scores that penalize low signal-to-noise regions) with SANE, a strategic sampling procedure that applies a non-uniform cost function to favor under-sampled locations over repeated visits to a known optimum. The whole loop is built on a deep kernel learning surrogate that maps structure images to spectroscopy, and the acquisition function decides the next measurement point.","core_discovery":"The central claim is that novelty scoring plus strategic sampling extends autonomous experiments from optimizing a known scalar descriptor to actively seeking unobserved phenomena. The paper's evidence comes from a ferroelectric dataset with known ground-truth domain structure: an experiment driven to maximize loop area quickly finds large-loop regions but overfits, ending with a normalized mean error (NME) of 0.30; experiments driven by Nearest Neighbor or Isolation Forest novelty scores instead keep sampling diverse regions and end at NME 0.15–0.16. Adding the SANE strategic sampler makes the experiment cluster measurements around local novelty maxima before jumping to distant regions, which initially raises NME (to 0.29–0.39) as it intentionally samples anomalous points, then drops to 0.16 by 200 measurements. In live scanning probe experiments on a ferroelectric film, both novelty-driven schemes raise the variability of collected hysteresis loops above the optimization-driven baseline, and the strategic variant shows a second variability jump at around step 30 attributed to finding a new region. The paper concludes that the novelty scoring system and the strategic sampling system act on different spaces—result space and parameter space, respectively—and contribute independently to discovery.","pith_inferences":["Beyond the paper's demonstrations, an adaptive controller that measures recent novelty yield and switches between pure novelty scoring and strategic sampling could outperform either fixed strategy, since the two mechanisms dominate at different phases of the run.","Because the five novelty metrics disagree about which regions are novel—for instance, one isolates domain walls while others highlight whole domains—the choice of metric encodes a physics prior; a principled way to select or combine metrics for a target material class would be a natural follow-up.","The strongest test the paper leaves open is whether high variability actually coincides with new physics: applying the same loop to a sample with independently mapped phases would settle whether the diversity gain is discovery or just noise."],"forward_implications":["Autonomous experiments aimed at novelty, not a fixed descriptor, can reconstruct the physical map more accurately at a fixed measurement budget: the novelty-driven runs end at NME 0.15–0.16 versus 0.30 for the scalarizer-driven run on the model dataset.","Strategic sampling makes the experiment temporarily worse by the optimization metric while it deliberately probes anomalous points, so practitioners should expect an NME rise before the payoff.","On the model dataset, novelty-driven sampling produces measured-loop variability above random sampling, while the optimization-driven run stays within random expectation, so novelty targeting is what drives diversity rather than the sheer number of samples.","The framework runs on a real microscope, not only on stored data, meaning the same acquisition loop can be deployed for live exploration of other materials."],"supporting_citations":[{"why":"Supplies the SANE strategic sampling mechanism whose non-uniform cost function drives exploration of under-sampled regions.","marker":"[42]"},{"why":"Supplies the implementations of four of the five novelty scoring methods used in the experiments.","marker":"[41]"},{"why":"Supplies the deep kernel learning methodology and the ferroelectric active-learning context that the acquisition loop builds on.","marker":"[23]"},{"why":"Describes the deep kernel learning surrogate model that maps structure images to spectroscopy in the acquisition loop.","marker":"[39]"},{"why":"Provides the autonomous scanning probe microscopy platform on which the live experiments were run.","marker":"[38]"},{"why":"Describes an earlier curiosity-driven framework that the paper positions its approach against and aims to complement.","marker":"[35]"},{"why":"Defines the physical-descriptor-driven autonomous experiment baseline that the novelty-driven methods are compared with.","marker":"[40]"}],"fun_headline_variants":["Novelty-guided autonomous experiments discover more diverse physics","INS2ANE: Novelty scoring beats optimization in autonomous labs","Autonomous experiments that seek novelty find unexpected phenomena","Novelty-driven exploration outperforms optimizers in autonomous experiments"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument rests on treating the spread of the measured polarization-response curves as evidence of new physics, without an independent ground-truth check that high-spread locations actually contain phenomena not present at low-spread locations.","fun_headline_variants_meta":{"raw":{"variants":["Novelty-guided autonomous experiments discover more diverse physics","INS2ANE: Novelty scoring beats optimization in autonomous labs","Autonomous experiments that seek novelty find unexpected phenomena","Novelty-driven exploration outperforms optimizers in autonomous experiments"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1350,"prompt_tokens":993,"completion_tokens":357,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":609,"completion_tokens_details":{"reasoning_tokens":291}},"tokens_in":609,"tokens_out":357,"duration_ms":4277,"temperature":1.0,"reasoning_tokens":291,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:47:55.284303+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test: on a sample with phases mapped by an independent structural method, run the three acquisition strategies and count how many previously unseen phase regions each one samples; the central claim fails if the novelty-driven and strategic runs do not find more such regions, or if their extra variability is dominated by noise rather than by distinct physical states.","supporting_citations":[{"cited_title":"SANE: strategic autonomous non-smooth exploration for multiple optima discovery in multi -modal and non- differentiable black-box functions,","cited_arxiv_id":null,"evidence_quote":"Supplies the SANE strategic sampling mechanism whose non-uniform cost function drives exploration of under-sampled regions."},{"cited_title":"Scikit -learn: Machine learning in Python,","cited_arxiv_id":null,"evidence_quote":"Supplies the implementations of four of the five novelty scoring methods used in the experiments."},{"cited_title":"Experimental discovery of structure –property relationships in ferroelectric materials via active learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the deep kernel learning methodology and the ferroelectric active-learning context that the acquisition loop builds on."},{"cited_title":"Automated experiments of local non -linear behavior in ferroelectric materials,","cited_arxiv_id":null,"evidence_quote":"Describes the deep kernel learning surrogate model that maps structure images to spectroscopy in the acquisition loop."},{"cited_title":"AEcroscopy: a software–hardware framework empowering microscopy toward automated and autonomous experimentation,","cited_arxiv_id":null,"evidence_quote":"Provides the autonomous scanning probe microscopy platform on which the live experiments were run."},{"cited_title":"Curiosity Driven Exploration to Optimize Structure-Property Learning in Microscopy,","cited_arxiv_id":null,"evidence_quote":"Describes an earlier curiosity-driven framework that the paper positions its approach against and aims to complement."},{"cited_title":"Scientific exploration with expert knowledge (SEEK) in autonomous scanning probe microscopy with active learning,","cited_arxiv_id":null,"evidence_quote":"Defines the physical-descriptor-driven autonomous experiment baseline that the novelty-driven methods are compared with."}],"review_version":1}