{"id":"7948a94d-a6ba-4046-b6ad-03d57a841771","arxiv_id":"2505.06519","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SHAP-guided bound refinement improves Bayesian optimization efficiency and final objective value for two underwater acoustic coating design problems.","lead":"This paper introduces an algorithm that uses SHAP, a machine learning interpretability tool, to automatically narrow the search space during Bayesian optimization. Applied to acoustic metamaterial coating design, it reports faster convergence and better final designs than standard Bayesian optimization.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All reported gains are computed on the prior DNN surrogate, not FEM; SHAP-bound refinement targets unvalidated extrapolation regions, so the 3%/11% improvements may not reflect real acoustic absorption.","rationale":"I read the paper as an algorithmic contribution plus an application: SHAP-based bound tightening is proposed to speed up BO, and the application is underwater acoustic coating design. The strongest quantitative claims are the speedup and the 3%/11% final improvements. The manuscript is honest about using the prior DNN surrogate as the objective to enable many repeated runs, and it reports means and standard deviations across seeds; the pseudocode is clear, and the authors flag the small-sample SHAP robustness concern in Section 3.3. These are real strengths. The condition that must hold for the material-design claim to be true is that the DNN surrogate's predictions are accurate in the regions the SHAP bounding selects. That condition is least secure because the method intentionally narrows the search to a 10% margin around the best-so-far point in the top-6 features, which may be outside the surrogate's training distribution. A high Pearson correlation from prior work does not establish pointwise accuracy in extrapolation, and no FEM evaluation is performed in this paper. I also noted that the 'off-the-shelf domain reduction' baseline and the standard BO settings are not specified and the number of seeds is not stated; these are secondary reproducibility concerns, but the surrogate validation is the more fundamental issue because even a perfect baseline comparison would not rescue a physical conclusion that is computed on an unvalidated model. The proposed FEM check would settle the concern directly; until it is done, the appropriate verdict remains CONDITIONAL, not ACCEPT or REJECT, because the method may well work and the evidence is suggestive but incomplete.","tokens_in":13436,"tokens_out":8284,"duration_ms":87197,"concrete_test":"Take the final mean-best designs from the SHAP-informed BO and classical BO runs for PU80 and PU90 (the top and bottom curves in Figs. 3c/4c), including their parameter vectors and DNN-predicted objective values. Evaluate these exact designs with the original FEM model from [11], computing the weighted absorption objective (Eq. 1) over the 10 Hz-10 kHz grid. Repeat for at least 10 random initial-sample seeds per method. If the FEM-evaluated objective for SHAP-informed designs is not positive relative to classical BO (or is materially smaller than the DNN-predicted 3%/11% gap), then the reported optimal solutions and speedup are surrogate artifacts rather than acoustic performance gains. This directly targets the unverified extrapolation region where the algorithm claims its advantage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Section 4: 'less than 1/3 of the iterations', '3% improvement... 11% improvement') is established entirely on the DNN surrogate from [11], not on the finite element model that defines the physical objective. Section 2.1 states explicitly: 'we use the surrogate model instead of directly executing new FEM simulations.' The DNN is reported to have a Pearson correlation of 0.999, but Pearson correlation does not control absolute error, and the SHAP-guided bound refinement (Algorithm 1, Section 2.3.3) deliberately moves the search into a 10%-margin neighborhood of the best-so-far point in the top-6 features. These neighborhoods can lie outside the distribution the DNN was trained on, precisely where a high in-distribution R2 is not evidence of accuracy. The convergence curves in Figs. 3c/4c and the final 'average' improvements are therefore DNN-predicted objective values. The authors themselves note in Section 3.3 that 'a sparse sampling of the parameter space could yield a misleading SHAP analysis' and that no robustness studies of SHAP on small data sets exist, but they do not validate the surrogate in the refined regions. If the surrogate is optimistic or biased there, SHAP-informed BO could appear to outperform classical BO while actually proposing worse or infeasible coatings. The missing FEM check is therefore load-bearing for the paper's physical-design claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SHAP-Bounded Bayesian Optimization, an algorithm that periodically applies SHAP analysis to a DNN surrogate trained on the data collected so far, then tightens the search bounds of the top-6 features according to whether all SHAP values to one side of the best-so-far point are positive. The method is applied to optimize ten geometric parameters of polyurethane acoustic metamaterial coatings for two materials, PU80 and PU90, using a DNN surrogate instead of finite element simulations to evaluate the objective. The reported results claim that SHAP-informed BO matches a 400-iteration classical BO result in fewer than one third of the iterations and yields final surrogate-objective improvements of about 3% for PU80 and 11% for PU90.","tokens_in":13791,"tokens_out":5360,"duration_ms":54895,"significance":"If the surrogate-level results transfer to the finite element model, the contribution is significant and timely: it is a simple, modular way to inject large-scale trend information into Bayesian optimization, it is clearly specified in Algorithm 1, and the paper provides a helpful toy illustration and repeated-run mean/standard-deviation convergence curves. The authors are also candid about several risk factors, including the possibility that sparse initial sampling may mislead SHAP. The current significance is limited, however, because the headline numbers are computed entirely on the DNN surrogate and because the main baseline is not specified; the paper is a promising algorithmic demonstration but not yet a validated material-design study.","major_comments":[{"comment":"The central quantitative claims ('less than 1/3 of the iterations', '3% improvement', '11% improvement') are computed on the DNN surrogate, not on the finite element model, and Section 2.1 explicitly states that the surrogate is used instead of new FEM simulations. Because Algorithm 1 moves the search into a 10%-margin neighborhood of the best-so-far point in the top-6 features, the refined regions can lie outside the distribution of the initial 100 samples that largely determine the DNN training set; a Pearson correlation of 0.999 does not bound absolute surrogate error there. The paper therefore does not yet demonstrate that the proposed designs improve real acoustic absorption. Please either validate the final designs, and ideally the refined regions, with FEM evaluations, or explicitly restrict all conclusions to surrogate-objective performance and revise the abstract and conclusion accordingly.","section":"Section 2.1; Figs. 3c and 4c"},{"comment":"The comparison method is described only as an 'off-the-shelf domain reduction technique applied after the first 100 random samples.' No name, implementation details, or reference are provided, so the reader cannot tell whether this is a standard baseline or a deliberately weak one. Specify the baseline algorithm precisely, cite a published version if one exists, and describe how its bound updates differ from the proposed SHAP-based updates.","section":"Section 3.3; Figs. 3c and 4c"},{"comment":"The bound-tightening rule depends on several hand-set quantities: the top-6 feature count, the 10% margin, the requirement that all SHAP values on one side of the best-so-far point be positive, and the update schedule at iterations 100, 150, and 200. No sensitivity analysis is reported for any of these choices, and Section 3.3 itself states that the reliability of SHAP on small datasets is an open question. The robustness claim is therefore not yet supported; at minimum, test the margin and update schedule over a few values and report how often the procedure excludes the best region.","section":"Algorithm 1; Section 2.3.3"},{"comment":"The manuscript reports mean curves with confidence intervals but never states the number of independent optimization runs, and the final '3%' and '11%' improvements are quoted without error bars or a statistical comparison. Without this information, the claim that the improvement is 'consistently achieved' cannot be evaluated; add the number of seeds and the dispersion of the final improvements for each method.","section":"Section 3.3; Figs. 3c and 4c"}],"minor_comments":[{"comment":"The first sentence says 'Figures 4 (a) and (b) present the results of the SHAP analysis for PU90,' but the section and figure concern PU80; this is a typo.","section":"Section 3.2"},{"comment":"In the paragraph beginning 'The Next curve up,' the text reads 'roughly 6% higher for for PU90 and 2% higher for for PU80'; the duplicated 'for' should be removed.","section":"Section 3.3"},{"comment":"The phrase 'refine the bounds of the optimization problem automatically automatically' contains a duplicated adverb; the abstract also states 'without increasing the number of simulation iterations,' which is redundant with a same-budget comparison and could be clarified.","section":"Abstract"},{"comment":"The statement 'All packages and models that supported this study are publicly available' conflicts with 'Source code may be obtained from the corresponding author upon reasonable request'; please clarify the actual availability policy.","section":"Replication of Results"},{"comment":"Reference [15] appears to duplicate Reference [11] with different formatting, and Reference [22] has 'n/a' for volume and pages; please check consistency and completeness of all entries.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the main risk is overstatement in the abstract and conclusion relative to the surrogate-only evaluation. The authors should be asked either to provide FEM validation of the proposed designs or to consistently frame the results as surrogate-objective improvements. The unspecified comparison baseline must also be described before the empirical claims can be assessed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know three things. First, the algorithmic idea is genuinely new as far as I can tell: SHAP has been used post hoc in materials optimization before, but the authors are the first to feed SHAP sign information back into the BO loop as an automatic bound-refinement rule. Second, the evaluation is entirely on a surrogate model, not on the FEM physics. Third, the paper is refreshingly candid about its own limitations, which makes the missing FEM check more frustrating rather than less.\n\nWhat it does well: Algorithm 1 is precisely specified, and the authors run repeated trials and report means and standard deviations. They compare against an off-the-shelf domain reduction baseline and also tried tuning acquisition functions and biasing factors, which is the right kind of diligence. The toy example in Fig. 1 is instructive and honestly labeled as crafted to illustrate plausibility. The discussion of why they chose a coarse positivity criterion, and the acknowledgment that sparse sampling could mislead SHAP, shows clear thinking.\n\nThe load-bearing soft spot is the surrogate-only evaluation. Section 2.1 explicitly states that the DNN surrogate from [11] is used instead of new FEM simulations. The reported 3% and 11% improvements are DNN-predicted objective values. A Pearson correlation of 0.999 does not control absolute error, and the SHAP-guided refinement moves the search into 10% margin neighborhoods of the best-so-far point—precisely the kind of extrapolation region where an in-distribution R² says little. The authors themselves note that no robustness studies of SHAP on small data exist, but they never validate the surrogate in the refined regions. So the acceleration claim may survive, but the physical-design claim is unsupported.\n\nMinor issues: the baseline \"off-the-shelf domain reduction\" is never named, which hurts reproducibility. The number of repeated runs is not reported, only means and SDs, and there are no significance tests. Code is available only \"upon reasonable request,\" not publicly. Fig. 4's caption says \"PU90\" where it should say \"PU80.\" References [15] and [18] appear to duplicate [11] and [14] with slightly altered author lists. These are minor but suggest a need for cleanup.\n\nThe central algorithmic idea is sound as a heuristic, and the paper is not circular: SHAP is trained on data already seen by the BO loop, but the gains are measured on subsequent held-out evaluations. The real question is whether the surrogate is trustworthy in the regions the method selects. That question is not answered here.\n\nWho this is for: researchers interested in interpretability-guided optimization, or in acoustic metamaterial design, will get a concrete heuristic and a fair discussion of its risks. It deserves serious peer review, but the revision should add at least a few FEM validation points in the refined regions, name the baseline, report run counts and significance tests, and release the code. I would not cite it in my own work until that validation exists.","headline":"A clear, honest description of a simple idea—use SHAP sign patterns to shrink BO bounds—but all reported gains are computed on a DNN surrogate, so the physical improvements are unproven.","tokens_in":14231,"tokens_out":2053,"would_cite":false,"duration_ms":22439,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SHAP-bounded Bayesian optimization matches classical BO in under a third of the iterations.","keywords":["Bayesian optimization","SHAP","interpretable machine learning","underwater acoustic metamaterials","sound absorption","domain reduction","inverse material design","deep neural network surrogate"],"falsifier":"Run the final SHAP-informed and standard-BO designs for both PU80 and PU90 through the original finite element model (or physical samples in an impedance tube) and compare the weighted absorption objective. If the surrogate's predicted objective differs materially from the FEM result, or if the SHAP-informed design is not better than standard BO under FEM evaluation, the claimed 3% and 11% improvements do not hold in the physical problem.","tokens_in":13255,"feed_emoji":"🌊","tokens_out":6859,"duration_ms":63420,"temperature":0.7,"pith_summary":"This paper claims that feeding SHAP interpretability analyses into Bayesian optimization can speed up and improve the optimization of underwater acoustic metamaterial coatings. The idea is to use the sign of SHAP values to tighten the allowed ranges of the most influential design variables, focusing the search where absorption is likely to be high. On two polyurethane materials, the authors report that the SHAP-informed routine matches a 400-iteration classical Bayesian optimization in fewer than a third of the iterations, and finishes the full budget with about 3% and 11% better objective values. The practical motivation is that each design evaluation is expensive, so any trustworthy way to reduce the search space matters.","feed_headline":"SHAP-bounded optimization beats plain BO in a third of the steps","feed_subtitle":"Applying SHAP sign information to tighten search bounds gives 3% to 11% better underwater acoustic coatings at the same budget.","key_machinery":"The central mechanism is the SHAP-guided bound-refinement loop. After an initial batch of 10 x D random samples, a deep neural network is trained and SHAP values are computed; for the top six features, if the best-so-far design has a positive SHAP value and all SHAP values to the right (or left) are positive, the lower (or upper) bound is moved to within 10% of the best-so-far value. The Bayesian optimizer then runs inside the tightened box using a Gaussian process surrogate and an expected-improvement acquisition function, and the loop repeats at iterations 100, 150, and 200. The SHAP analysis supplies large-scale trend information that a Gaussian process with a stationary kernel tends to miss, while BO supplies the local search.","core_discovery":"The central claim is that SHAP-bounded Bayesian optimization, where SHAP values computed from a deep neural network surrogate are used to automatically tighten the design-space bounds, accelerates Bayesian optimization and yields better final designs than standard Bayesian optimization and a generic domain-reduction baseline. Applied to the ten-dimensional design of polyurethane coatings with embedded voids for underwater sound absorption, the method identifies the most influential geometric parameters, excludes regions with consistently negative SHAP values, and repeatedly re-bounds the search every 50 iterations after the 100th. In repeated runs with varied initial samples, SHAP-informed BO matched classical BO's 400-iteration result within about 130 iterations for both PU80 and PU90, and by the end of the same budget achieved on average 3% (PU80) and 11% (PU90) higher objective values. The authors present this as the first algorithmic integration of SHAP into the BO loop, as opposed to using SHAP only for post hoc analysis.","pith_inferences":["Editorial inference: the same signed-SHAP bound tightening should transfer to other expensive black-box design problems whose objectives show monotonic large-scale trends in at least a few parameters, such as structural or optical design, though the paper only demonstrates it on acoustic coatings.","Editorial inference: because the 3% and 11% improvements are computed on the DNN surrogate rather than on new finite element simulations, re-evaluating the final designs in the FEM or in experiments is needed to confirm the physical gain; the paper does not report such validation.","Editorial inference: the plateau after each bound refinement suggests an adaptive stopping rule, halt when no bounds change for a fixed number of iterations, which the paper mentions as a possibility for early stopping but does not implement."],"forward_implications":["With a 400-iteration budget as the reference, SHAP-informed BO reaches the same objective in under one third of the iterations, which translates into a comparable reduction in expensive simulations or experiments.","Repeated SHAP-bound applications yield higher final objectives at equal budget: about 3% for PU80 and 11% for PU90 over standard BO.","Because each bound refinement is followed by a rapid improvement and then a plateau, runs can be stopped early; the paper notes the top curve plateaus by iteration 300, saving 25% of the budget without losing the 11% gain.","The bound-refinement step is wrapped around the BO loop rather than replacing it, so it can be combined with any Gaussian-process kernel or acquisition function.","The method adapts to the material: the sets of influential parameters differ between PU80 and PU90, meaning SHAP-bounding is not applying one fixed reduction but is guided by each material's response surface."],"supporting_citations":[{"why":"Defines SHAP and its theoretical guarantees (local accuracy, consistency), the interpretability tool the whole bound-refinement step is built on.","marker":"[1]"},{"why":"Supplies the experimentally validated finite element model, the two polyurethane materials, and the deep neural network surrogate (Pearson correlation 0.999) that all optimization runs in this paper evaluate.","marker":"[11]"},{"why":"Demonstrates SHAP-based interpretation of a surrogate model in the authors' earlier work, which motivates using SHAP on the DNN in the optimization loop.","marker":"[2]"},{"why":"Shows SHAP used for knowledge discovery from surrogate models, providing the conceptual basis for converting SHAP insight into design-space narrowing.","marker":"[23]"},{"why":"Describes DeepLIFT, the gradient-based attribution method that Deep SHAP combines with Shapley values to make SHAP feasible for the DNN.","marker":"[32]"},{"why":"Benchmarks Bayesian optimization across materials-science domains, establishing BO as the baseline optimizer that this paper accelerates.","marker":"[16]"}],"fun_headline_variants":["SHAP-bounded BO matches 400-step result in 130 steps","SHAP-guided optimization cuts acoustic design iterations by two-thirds","SHAP-informed bounds yield up to 11% better acoustic coatings at same cost","Interpretable BO speeds up metamaterial coating design without extra simulations","SHAP-bounded Bayesian optimization: faster, better, and explainable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole comparison assumes the deep neural network surrogate faithfully reproduces the true acoustic absorption of the finite element model across the full design space, because every optimization run and every reported improvement is evaluated on that surrogate rather than on new simulations.","fun_headline_variants_meta":{"raw":{"variants":["SHAP-bounded BO matches 400-step result in 130 steps","SHAP-guided optimization cuts acoustic design iterations by two-thirds","SHAP-informed bounds yield up to 11% better acoustic coatings at same cost","Interpretable BO speeds up metamaterial coating design without extra simulations","SHAP-bounded Bayesian optimization: faster, better, and explainable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000676,"raw_usage":{"total_tokens":3083,"prompt_tokens":961,"completion_tokens":2122,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":2029}},"tokens_in":577,"tokens_out":2122,"duration_ms":18237,"temperature":1.0,"reasoning_tokens":2029,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:40:10.364723+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the final SHAP-informed and standard-BO designs for both PU80 and PU90 through the original finite element model (or physical samples in an impedance tube) and compare the weighted absorption objective. If the surrogate's predicted objective differs materially from the FEM result, or if the SHAP-informed design is not better than standard BO under FEM evaluation, the claimed 3% and 11% improvements do not hold in the physical problem.","supporting_citations":[{"cited_title":"In: Guyon, I.e.a","cited_arxiv_id":null,"evidence_quote":"Defines SHAP and its theoretical guarantees (local accuracy, consistency), the interpretability tool the whole bound-refinement step is built on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the experimentally validated finite element model, the two polyurethane materials, and the deep neural network surrogate (Pearson correlation 0.999) that all optimization runs in this paper evaluate."},{"cited_title":"Journal of Rheology69(2), 183–199 (2025) https://doi.org/ 10.1122/8.0000930","cited_arxiv_id":null,"evidence_quote":"Demonstrates SHAP-based interpretation of a surrogate model in the authors' earlier work, which motivates using SHAP on the DNN in the optimization loop."},{"cited_title":"In: AIAA SciTech Forumv, National Harbor, United States, p","cited_arxiv_id":null,"evidence_quote":"Shows SHAP used for knowledge discovery from surrogate models, providing the conceptual basis for converting SHAP insight into design-space narrowing."},{"cited_title":"In: Proceedings of the 34th International Con- ference on Machine Learning - Volume 70","cited_arxiv_id":null,"evidence_quote":"Describes DeepLIFT, the gradient-based attribution method that Deep SHAP combines with Shapley values to make SHAP feasible for the DNN."},{"cited_title":"npj Computational Materials7(1), 188 (2021) https://doi.org/10.1038/ s41524-021-00656-9","cited_arxiv_id":null,"evidence_quote":"Benchmarks Bayesian optimization across materials-science domains, establishing BO as the baseline optimizer that this paper accelerates."}],"review_version":1}