{"id":"775a259b-cbf3-472e-85d6-f8fe25022e1d","arxiv_id":"2412.17675","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"hep-aid is a modular Python library that packages active search, neural network, and MCMC parameter scan methods with a Higgs physics software stack, and its demonstrations show sample efficiency gains on test and BSM examples.","lead":"This paper presents hep-aid, a Python library for running machine-learning-driven parameter scans in high-energy physics models beyond the Standard Model. The library wraps active search, neural network, and MCMC methods with standard particle physics tools, and the authors report that it finds viable parameter regions with fewer evaluations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 95% efficiency claim lacks a baseline: without the satisfactory fraction of the initial 400 random points and a random-sampling comparison, the central efficiency claim is not yet supported.","rationale":"The reader's weakest-assumption analysis identifies exactly the load-bearing issue: the efficiency conclusions are drawn from one 2D test function and one BSM example, with no independent benchmark, no random or grid baseline, and no error bars. My stress-test converges on the same point, sharpened to a concrete quantity: the satisfactory fraction of the initial 400-point dataset. If that fraction is already high, the reported 95% Sr is largely a property of the constraints and initial sampling rather than of b-CASTOR or hep-aid. The paper is honest about being a manual and provides public code, which is real evidence of functionality, but the central claim in the abstract and conclusions is a performance claim, and performance claims need baselines. My proposed check would settle the concern in one experiment: compare the active-search Sr against the initial-dataset Sr and against random sampling with the same budget. I therefore recommend keeping the reader's CONDITIONAL verdict unchanged, with the condition being the addition of a baseline comparison and uncertainty quantification before the efficiency claim is accepted at face value.","tokens_in":18637,"tokens_out":2312,"duration_ms":25383,"concrete_test":"Re-run the (B-L)SSM example with the exact configuration of Section 2.2, and first record the satisfactory fraction Sr of the initial 400-point random dataset before any active-search iterations. Then run a uniform random (or quasi-Monte Carlo) sampler to the same total number of SPheno evaluations with identical constraints and δm = 5 GeV, over at least 10 seeds, and compare mean Sr and its spread. If random sampling reaches Sr comparable to b-CASTOR's ~95%, the efficiency enhancement claim is unsupported; if random sampling is substantially lower, the claim survives this test. Additionally, report b-CASTOR's Sr and coverage over 10 seeds to quantify run-to-run variability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that hep-aid methodologies 'enhance the efficiency of BSM studies.' In Section 2.2, this rests on a single (B-L)SSM demonstration in which b-CASTOR reaches 'nearly 95% efficiency,' measured as Sr = satisfactory/total samples, with 400 initial points and a 5 GeV mass window. The problem is that Sr is not an algorithm-intrinsic quantity: it depends on the initial dataset, the constraint width, and the sampling policy's tendency to query near already-satisfactory regions. The paper never reports the satisfactory fraction of the initial 400-point random dataset, so the reader cannot tell how much of the 95% is due to the active search and how much is inherited from the starting set. If a comparable uniform random sampler with the same evaluation budget and the same constraints also achieves a high Sr, then the reported enhancement is not demonstrated. The comparison on the 2D test function in Figure 8 also lacks a random or grid baseline, and the BSM run has no error bars or multiple-seed variance. The library itself appears functional and well-documented, but the load-bearing performance claim is supported by narrow, self-referential evidence from the authors' earlier paper [16] rather than an independent benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents hep-aid, a modular Python library for sample-efficient parameter scans in BSM phenomenology. It describes the library's two main modules — the hep module (interfacing SPheno, HiggsBounds, HiggsSignals, and MadGraph) and the search module (providing an Objective class and four parameter-scan methods: CAS, b-CASTOR, MLScan, and MCMC-MH) — along with configuration files, checkpointing, SLHA handling, and plotting utilities. The paper demonstrates the library on a two-dimensional double-objective test function proposed in the authors' earlier work and on an 8D (B-L)SSM scan targeting a ~95 GeV scalar together with the 125 GeV Higgs state. The central claim is that the included ML-based search methods 'enhance the efficiency of BSM studies', supported by an efficiency comparison among the methods and a reported 'nearly 95% efficiency' for b-CASTOR in the BSM example.","tokens_in":18938,"tokens_out":6533,"duration_ms":56422,"significance":"If the efficiency claim were properly supported, hep-aid would be a valuable community tool: it addresses a real need for modular, ML-based scan infrastructure that connects to standard spectrum generators, and it ships with several search algorithms, configuration-driven workflows, checkpointing, and dataset utilities. The paper is transparent about the library's design, provides code examples, and indicates availability of source code and reproducibility resources. However, the central performance claim currently rests on narrow, self-referential demonstrations that lack baselines and error bars, so the paper's significance hinges on whether the authors add an independent benchmark in a revision.","major_comments":[{"comment":"The comparison of b-CASTOR, MLScan, and MCMC-MH reports only the mean efficiency Sr over runs (five runs for b-CASTOR and MLScan, ten for MCMC-MH) and displays no error bars, quantiles, or individual run trajectories. The text itself notes that MCMC-MH achieved Sr=0 in some runs, which indicates substantial run-to-run variability, but the figure cannot convey this. More importantly, no uniform random-sampling or grid baseline is included; without a baseline, the reader cannot determine how much of Sr is attributable to the active search strategy rather than to the ease of the test problem or the choice of constraints.","section":"Section 2.1.2, Figure 8"},{"comment":"The claim that b-CASTOR achieves 'nearly 95% efficiency' in the (B-L)SSM scan is not a property of the algorithm alone: it depends on the initial dataset of 400 points, the user-defined mass window δm = 5 GeV, the total evaluation budget, and the number of repeated runs. The paper does not report the satisfactory fraction of the initial 400 random points, nor does it compare b-CASTOR against uniform random sampling with the same constraints and budget. Because Sr is the ratio of satisfactory to total sampled points, the reported 95% could be largely inherited from a favorable initial set; the claim that the method 'enhances' efficiency therefore needs control measurements to be supported.","section":"Section 2.2, Eq. (3) and Figure 10"},{"comment":"The conclusion states that 'All ML-based such approaches were also demonstrated to be superior to the more standard MCMC-HS one' and the abstract asserts that demonstrations 'show that hep-aid methodologies enhance the efficiency of BSM studies.' These generalisations are based on a two-dimensional test function and a single 8D BSM example, with the BSM example lacking multiple-seed variation and both lacking an independent baseline. The strength of the conclusion exceeds the evidence presented; it should be restricted to the specific test cases or supported by a broader benchmark suite.","section":"Section 5 and Abstract"},{"comment":"The evaluation is self-referential in a way that weakens external validity: the 2D test function, the (B-L)SSM case study, and the b-CASTOR algorithm all originate from the authors' earlier paper [16], and the comparison methods MLScan and MCMC-MH are used as implemented in the library itself. This is not a logical circularity, but it means the benchmark does not test the library against independently developed methods or problems. Adding at least one independent test function (e.g., from a public black-box optimisation benchmark) and one independent BSM model would substantially strengthen the claims of versatility and sample efficiency.","section":"Section 2.1.2, Section 2.2, and reference [16]"}],"minor_comments":[{"comment":"The sentence 'However, in hep-aid implements its own SLHA, module.lass' appears to contain a typo; it should likely read 'hep-aid implements its own SLHA module, with the SLHA class.'","section":"Section 3.3"},{"comment":"The axis label rendered as 'D∫⟩‡⌉' is corrupted; it presumably should be 'D_size' or 'Dataset size'.","section":"Figure 8"},{"comment":"The phrase 'the more standard MCMC-HS one' uses an acronym inconsistent with the rest of the paper, where the method is defined as MCMC-MH.","section":"Section 5"},{"comment":"In the sentence 'Another notable library is [45]', the library is not named at the point of citation; naming it (xSLHA) immediately would improve readability.","section":"Section 3.2"},{"comment":"The installation instructions do not specify the tested Python version or the versions of key dependencies; stating the exact environment would aid reproducibility.","section":"Section 2.1.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is essentially a software manual, and as such the performance benchmarking is the weakest part. The authors should be encouraged to add random-sampling baselines, multiple-seed error bars, and an independent benchmark in the revision; the library itself appears well-structured and potentially useful. The heavy reliance on the authors' own earlier work [16] for the algorithm, test function, and case study should be made transparent to the reader."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: hep-aid is a real, usable Python library for BSM parameter scans, and the paper is a good manual for it. The abstract's claim that these methods 'enhance the efficiency of BSM studies' is, however, not yet backed by the evidence in the paper.\n\nWhat's new: the library itself. It wraps SPheno, HiggsBounds, HiggsSignals and MadGraph into a clean Python API, and provides a config-driven framework where you can plug in different search algorithms—CAS, b-CASTOR, MLScan, and MCMC-MH. The modular design, the SLHA handling, and the checkpointing/dataset I/O are all sensible and well illustrated with code snippets. The tutorial-style presentation makes it easy to get started, and the code is on GitHub, so people can actually run it.\n\nThe soft spots are all around the performance demonstration. Figure 8 reports means over five to ten runs but shows no error bars or spread, and there's no random or grid sampling baseline. The 2D test function comes from the authors' own previous paper [16], and so does b-CASTOR, so the benchmark is self-referential. The BSM example is a single (B-L)SSM fit with a 400-point initial dataset and a 5 GeV mass window. They report 'nearly 95% efficiency' (Sr = satisfactory/total), but they never tell you what fraction of those initial 400 random points were already satisfactory. Without that number, you can't separate the active search's contribution from the luck of the initial sample. Sr also depends on the constraint width and batch size, so 95% isn't a fixed property of the algorithm.\n\nNone of this means the library is bad. It means the headline claim is ahead of the evidence. The authors can fix it by adding a random-sampling comparison, error bars on the multiple runs, reporting the initial satisfactory fraction, and ideally benchmarking against a generic MCMC or one of the existing tools like BSMart or xBit on a shared test case. The paper would then be honest about what the library does well and where it still needs work.\n\nWho should read this: BSM phenomenologists who want a modular Python tool for scans, and people who develop scan algorithms and want a clean testbed. It's the kind of software paper that earns a serious referee: the artifact is real, the documentation is good, and the weaknesses are in the evaluation, not in the code. I'd send it to peer review, with requests for the additional benchmarks and a toned-down abstract.","headline":"A genuinely useful and well-documented software library, but the headline efficiency claim needs a random baseline and error bars before I'd believe it.","tokens_in":19449,"tokens_out":3044,"would_cite":true,"duration_ms":28890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents hep-aid, a modular Python library for machine-learning-guided, sample-efficient parameter scans in beyond-Standard-Model phenomenology, and reports that its b-CASTOR method reaches nearly 95% efficiency on a (B-L)SSM…","keywords":["hep-aid","parameter scan","active search","b-CASTOR","beyond Standard Model","Higgs mass","machine learning","surrogate model"],"falsifier":"Run b-CASTOR and a uniform random sampler on the same (B-L)SSM objective, at the same evaluation budget and with the same initial dataset; if the active-search ratio of satisfactory to total points is not higher than random sampling's, the central efficiency claim fails.","tokens_in":18455,"feed_emoji":"🎯","tokens_out":8982,"duration_ms":88872,"temperature":0.7,"pith_summary":"hep-aid is a Python library that packages machine-learning-guided parameter scans for beyond-Standard-Model (BSM) phenomenology, with the goal of finding parameter regions that satisfy experimental constraints using as few expensive spectrum evaluations as possible. The paper argues that the active-search methods it implements, especially b-CASTOR, a batched variant of constraint active search, are more sample-efficient than a Metropolis-Hastings MCMC baseline. On a two-dimensional test function b-CASTOR shows the highest ratio of satisfactory points, and on an eight-dimensional (B-L)SSM scan targeting a 95 GeV Higgs-like state alongside the 125 GeV SM-like Higgs, it reaches nearly 95% efficiency with an initial dataset of 400 points. The library is structured so that new scan algorithms and new HEP software can be plugged in through configuration files.","feed_headline":"hep-aid finds satisfactory BSM points in ~95% of samples","feed_subtitle":"Machine-learning active search targets the 95 GeV and 125 GeV Higgs windows with fewer costly spectrum evaluations.","key_machinery":"The load-bearing mechanism is the active-search loop. An Objective object wraps a black-box function $H(x)$, a set of constraints $\\tau$, and an internal dataset; at each iteration a Gaussian-process surrogate is fitted to the normalised dataset, and the Expected Coverage Improvement (ECI) acquisition function scores candidate points by how much a hypersphere of radius $r$ placed there would expand coverage of the satisfactory region $S = \\{x \\mid H_i(x) \\succeq \\tau_i\\}$. b-CASTOR is the batched version: it optimises ECI with the Tree-structured Parzen Estimator and then uses stochastic prioritisation to pick a batch of points each iteration. The library implements CAS, b-CASTOR, MLScan (an MLP surrogate with rejection sampling), and MCMC-MH as a comparison baseline, all sharing the same Objective, dataset, checkpoint, and metric machinery.","core_discovery":"The paper's central claim is that hep-aid's active-search methodologies make BSM parameter scans sample-efficient: instead of evaluating every candidate with the full HEP software stack, a Gaussian-process surrogate is fitted to collected points and an acquisition function decides which parameter vectors to evaluate next. For the (B-L)SSM case study, with the objective space reduced to the two lightest Higgs mass observables and the constraints set to $95 \\pm 5$ GeV and $125 \\pm 5$ GeV, b-CASTOR reports a satisfactory-to-total ratio $S_r \\approx 0.95$ starting from 400 initial points. On the double-objective test function from [16], b-CASTOR outperforms MLScan and MCMC-MH on this efficiency metric, while MLScan shows broader exploration across the parameter space. The paper concludes that these results support the library as a versatile toolset for multi-objective searches for new physics.","pith_inferences":["Because the efficiency metric $S_r$ depends on the width of the mass window and the size of the initial dataset, the ~95% figure is probably specific to the 5 GeV window used here rather than a fixed property of b-CASTOR.","A direct comparison against a uniform random sampler on the same (B-L)SSM objective would separate the contribution of the surrogate policy from the ease of the problem; the paper does not report such a baseline.","The same Objective-plus-surrogate-plus-acquisition structure could be adapted to other expensive black-box searches in high-energy physics, such as event-generator tuning or detector simulation parameters, with only configuration changes.","The ECI radius decay, currently a fixed schedule, could be tied to the surrogate's predictive uncertainty, which might make the scan adaptive in higher-dimensional spaces."],"forward_implications":["If b-CASTOR's high hit rate holds across runs, BSM scans can concentrate expensive spectrum evaluations inside satisfactory regions instead of spreading the budget across the whole parameter box.","A new BSM model can be scanned by swapping the model files and configuration while reusing the same active-search policy and dataset machinery.","The library's automatic metric tracking makes it straightforward to compare CAS, b-CASTOR, MLScan, and MCMC-MH on the same objective in a few lines of code.","Checkpointing and compressed dataset export allow long scans to be resumed or shared, supporting reproducible phenomenological studies."],"supporting_citations":[{"why":"Supplies the b-CASTOR algorithm, the double-objective test function, and the (B-L)SSM example that grounds the reported 95% efficiency.","marker":"[16]"},{"why":"Defines the Constraint Active Search algorithm that b-CASTOR batches and that hep-aid implements.","marker":"[33]"},{"why":"Defines the MLScan method, the neural-network rejection-sampling scan bundled in the library.","marker":"[7]"},{"why":"Establishes the active-search paradigm that the Objective, surrogate, and policy loop implements.","marker":"[23]"},{"why":"Provides the SPheno spectrum calculator that produces the Higgs mass observables in the HEP-stack.","marker":"[28]"},{"why":"Provides HiggsBounds, used to test BSM Higgs predictions against experimental exclusion limits.","marker":"[30]"},{"why":"Provides HiggsSignals, used to compare model Higgs sectors with the measured 125 GeV state.","marker":"[31]"},{"why":"Reports the CMS di-photon excess near 95 GeV that motivates the mass constraints in the case study.","marker":"[27]"},{"why":"Provides the Metropolis-Hastings sampling baseline used for the efficiency comparison.","marker":"[34]"}],"fun_headline_variants":["hep-aid: 95% sample-efficient BSM scans","Active search finds BSM points with 95% success","ML active search boosts BSM scan efficiency to 95%","Sample-efficient BSM scans with hep-aid","Python library hits 95% success in BSM scans"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The broad efficiency conclusion rests on one two-dimensional test function and one eight-dimensional (B-L)SSM scan with a 5 GeV mass window and a 400-point initial dataset, so the reported hit rate may not transfer to other models, objectives, or baselines.","fun_headline_variants_meta":{"raw":{"variants":["hep-aid: 95% sample-efficient BSM scans","Active search finds BSM points with 95% success","ML active search boosts BSM scan efficiency to 95%","Sample-efficient BSM scans with hep-aid","Python library hits 95% success in BSM scans"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001141,"raw_usage":{"total_tokens":4766,"prompt_tokens":1007,"completion_tokens":3759,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":3677}},"tokens_in":623,"tokens_out":3759,"duration_ms":23773,"temperature":1.0,"reasoning_tokens":3677,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:15:43.689844+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run b-CASTOR and a uniform random sampler on the same (B-L)SSM objective, at the same evaluation budget and with the same initial dataset; if the active-search ratio of satisfactory to total points is not higher than random sampling's, the central efficiency claim fails.","supporting_citations":[{"cited_title":"Malkomes, B","cited_arxiv_id":null,"evidence_quote":"Defines the Constraint Active Search algorithm that b-CASTOR batches and that hep-aid implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MLScan method, the neural-network rejection-sampling scan bundled in the library."},{"cited_title":"Garnett, Y","cited_arxiv_id":null,"evidence_quote":"Establishes the active-search paradigm that the Objective, surrogate, and policy loop implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SPheno spectrum calculator that produces the Higgs mass observables in the HEP-stack."},{"cited_title":"Bechtle, S","cited_arxiv_id":null,"evidence_quote":"Provides HiggsSignals, used to compare model Higgs sectors with the measured 125 GeV state."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports the CMS di-photon excess near 95 GeV that motivates the mass constraints in the case study."}],"review_version":1}