{"id":"027fb6fa-a662-4ede-a654-16abe7dc42cb","arxiv_id":"2607.21220","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SHA-PF uses initial simulation data to select a 'hard but promising' anchor satisfaction state and evolves LLM-generated formulations that prioritize it, reaching target designs with fewer expensive simulations on antenna and hydrology benchmarks.","lead":"SHA-PF is an LLM-based system that turns natural-language design goals into optimization formulations chosen to steer the optimizer along rare, promising paths in the initial data. On six expensive real-world benchmarks (one hydrology, five antennas) it reaches requirement-satisfying designs in fewer simulations than expert or direct-LLM formulations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Anchor loss is selected and evaluated on the same 100 initial samples, so the reported efficiency gain may reflect selection on the evaluation data rather than a general property of SHA-PF formulations.","rationale":"The reader's weakest_assumption is that 100 initial space-filling evaluations reveal the search-hardness structure, and that anchor selection is reliable. My concern is more specific and sharper: even if the 100 samples do reveal structure, the formulation objective (Eq. 7) is measured on exactly those samples, so the selection step itself may be overfitting to D_init. This is a correctness risk rather than a disagreement with consensus, because the paper presents no variance analysis, no repeated runs, and no held-out validation. The internal evidence (ablations, cross-optimizer checks, backbone robustness) supports the mechanism but does not address selection-on-evaluation-data. I agree with the reader's assessment that the paper is conditionally acceptable; the missing repeated-initialization study is the single check that would settle whether the central claim generalizes. I do not see a separate, more load-bearing concern: the 200/300 budget inconsistency and the single-run comparisons are real but secondary to the in-sample selection issue.","tokens_in":26624,"tokens_out":1483,"duration_ms":15521,"concrete_test":"Resample the initialization: for the HSE benchmark (23 variables, 41 response samples, cheapest at 511 s), draw, say, 10 independent Latin-hypercube D_init sets of size 100; rerun the full SHA-PF formulation search and the final 200-evaluation SCBO search for each D_init. If the formulation found under the original D_init wins, on average, over the other D_init-restarts and also beats the baseline formulations under all D_init sets, the in-sample-selection concern is resolved. A cheaper analytical check: for a fixed family of plausible anchors, compute the rank correlation between Eq. (7) scores on D_init and Eq. (7) scores on disjoint held-out initial samples from the same space; low correlation would indicate the objective is dominated by sampling noise.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that Eq. (7) selects formulations that require significantly fewer evaluations because they prioritize rare, promising anchor states. But the anchor state z_anc is chosen from Eq. (5) using the same 100-sample D_init that is then used to score every candidate formulation in Eq. (7), and the same D_init is also used by all baselines as the starting population of the final expensive search (Table 1 'excluding the initial evaluations'). The formulation search therefore optimizes an objective on the very data that define the benchmark's success metric. Nothing in the paper shows the selected anchor or the top-scoring formulation is stable under resampling: if a different D_init yields a different z_anc and a materially different winning formulation, then the advertised 'significantly fewer evaluations' is partly an artifact of in-sample selection on D_init. The paper's own Conclusion admits anchor selection 'may be less reliable when these samples are sparse or miss hard yet promising regions,' which is precisely the regime of 100 samples in high-dimensional (13–23 variable) antenna spaces. The reader's weakest_assumption identifies exactly this dependence on initial samples; the concern is load-bearing because without a held-out demonstration, Eq. (7) could be selecting formulations that happen to rank the anchor high on those 100 points while providing no general search-efficiency advantage.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SHA-PF, a framework that uses an LLM to search over alternative optimization formulations for expensive simulation-driven design. Given an initial set of 100 evaluations D_init, SHA-PF (i) computes a search-hardness score h(z) for each observed binary satisfaction state (Eq. 4), (ii) selects an anchor state z_anc by minimizing h(z) over non-dominated states (Eq. 5), and (iii) scores candidate formulations by the fraction of anchor-vs-other response pairs in which the anchor response ranks higher under the formulation-induced ordering (Eq. 7). The formulations are generated, repaired, and evolved using LLM prompts. The paper reports experiments on the HBV hydrological benchmark and five antenna design tasks, claiming that SHA-PF finds requirement-satisfying designs with significantly fewer evaluations than expert-written or directly LLM-generated formulations.","tokens_in":26931,"tokens_out":4823,"duration_ms":57065,"significance":"If the central claim were established, SHA-PF would be a useful contribution: it addresses a real gap in automated problem formulation by considering search efficiency rather than only requirement alignment. The method is clearly described, the generation/repair/refinement pipeline is sensible, and the inclusion of cross-optimizer and LLM-backbone sensitivity analyses is a strength. However, the empirical evidence as presented does not support the advertised 'significantly fewer evaluations' claim. The results are single runs, the formulation objective is optimized on the same data used to define the benchmark's success, and the reported time budgets are internally inconsistent. The central finding — that prioritizing rare, promising anchor states in the initial data yields formulations that generalize to more efficient search — is asserted rather than demonstrated on held-out data. With additional experiments involving resampling and out-of-sample evaluation, the claim could become credible, but the current version is not.","major_comments":[{"comment":"Every cell in Tables 1 and 2 is a single run, with no variance, no repetitions, and no statistical test. The abstract and text repeatedly say 'significantly fewer evaluations' (e.g., abstract; §Main Results, 'SHA-PF succeeds on all five antenna tasks'), but Table 1 provides only point estimates. Since SHA-PF involves stochastic LLM generation/repair and random initial samples, and the baselines are also stochastic, single runs cannot support a significance claim. Please report multiple independent repetitions with different D_init and LLM sampling, and provide medians, IQRs, and a paired test or effect-size measure.","section":"Main Results, Tables 1–2"},{"comment":"The formulation search objective Eq. (7) maximizes anchor-vs-other priority computed on D_init, and the anchor z_anc is itself selected from the same D_init via Eqs. (4)–(5). Moreover, Table 1's caption confirms that the final expensive search for all methods also starts from the same D_init ('excluding the initial evaluations'). Thus a formulation that overfits D_init can score high on Eq. (7) and simultaneously bias the starting population of the final search, without providing any general search-efficiency benefit. The paper's own Conclusion admits that anchor selection 'may be less reliable when these samples are sparse or miss hard yet promising regions.' This is exactly the regime of 100 samples in 13–23 dimensional antenna problems. Please add a held-out or cross-validated protocol: select the formulation on D_init_train and evaluate it on a freshly sampled D_init_test, or repeat","section":"Formulation Search Objective, Eq. (7)"},{"comment":"There is an internal inconsistency in the evaluation budget. The main text states 'For each antenna design task, the budget is 300 HFSS simulations,' but Table 3's note says the sensitivity experiments use 'a budget of 200 evaluations.' Additionally, the reported time costs in Table 1 cannot be reconciled with the average per-simulation times in Table 8. For example, HAE has an average evaluation time of 532 s, so 66 evaluations would require about 9.7 h, yet Table 1 reports 2.7 h for SHA-PF (SCBO); for CSL, 10 evaluations at 963 s would be about 2.7 h, but Table 1 reports 2.1 h. The discrepancy factor varies across tasks, suggesting unreported parallelism or a different notion of time. Please clarify the evaluation budget and define exactly how 'hours' is measured; otherwise the time savings claims are not reproducible.","section":"Experimental Setup and Table 3"},{"comment":"The paper's central finding — 'a formulation is more likely to guide efficient search when it prioritizes rare samples with greater progress potential' — is embedded in the method rather than tested. The ablation replacing anchor selection with random anchor selection (Table 2) compares against the proposed selection on the same D_init, but it does not test alternative hardness definitions or demonstrate that the specific h(z) criterion is the reason for success. To make the claim load-bearing, please add a controlled comparison on held-out D_init: e.g., select formulations by maximizing Eq. (7) with the proposed anchor versus with a random anchor versus with rarity-only or progress-only criteria, and show that the proposed anchor yields better search efficiency on fresh data, not just on the D_init used for selection.","section":"Finding: Initial Data Reveal Search Hardness; Ablation Study"}],"minor_comments":[{"comment":"The claim that SHA-PF 'is not tied to a specific optimizer' would be stronger if the baseline formulations were also evaluated with DSI and CEBO, since the current comparison only shows that one SHA-PF formulation works with SCBO, DSI, and CEBO to differing degrees. Without baseline results for the same optimizers, the conclusion that 'optimizer choice is important but not the main reason for the performance gain' is not directly supported.","section":"Cross-Optimizer Evaluation"},{"comment":"In Eq. (4), the denominator notation |{i|z≻z_i}| is slightly ambiguous because z_i ranges over observed states; writing |{i ∈ {1,...,N} : z ≻ z_i}| would be clearer.","section":"Equation (4) notation"},{"comment":"The text defines Φ_R as the set of requirement-aligned formulations, but the repair step only 'aims to move' the candidate toward Φ_R, so the actual candidate set Φ̂_R used in Eq. (7) may contain formulations that are not truly requirement-aligned. Please clarify this mismatch and state whether any post-hoc verification is performed before the final optimization.","section":"Section 3.3, Φ_R vs. Φ̂_R"},{"comment":"In Figure 5(a), the x-axis appears to end at generation 12, while the text and Appendix A use G_max = 25. If the plot is truncated, this should be stated; otherwise the curves should extend to G_max.","section":"Figure 5"},{"comment":"There are several typos and awkward phrasings, e.g., 'The formulation is selected using same initial data that seed the reported' (if present in the final version), and 'allowing more generations provides additional opportunities to improve such populations and reach the maximum objective value.' A light editorial pass would improve readability.","section":"Language"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the method is coherent, but the experimental validation is not yet at the level needed for the strong claims made. The single-run results, the in-sample selection of Eq. (7) on the same D_init used in evaluation, and the unexplained time/budget inconsistencies all need to be addressed with additional experiments and a clarified protocol. I would also encourage the authors to release code and the discovered formulations for reproducibility. This is a major-revision rather than a reject because the identified problems are fixable within the scope of the paper: resampling-based evaluation and a held-out anchor-selection test are standard additions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper has a real new idea—formulation quality treated as a search-efficiency issue, not just correctness—and the authors actually ship the formulations they used, which is refreshing. The anchor-state criterion (rare but promising states get priority) is coherent, and the ablations and cross-optimizer checks give some confidence the mechanism is doing the work.\n\nThe soft spots are in the empirical evidence. Every benchmark result appears to be a single run. The abstract says \"significantly fewer evaluations\" but there is no variance, no error bars, no significance test anywhere. That needs to be fixed with repeated runs and proper statistics. The in-sample issue is also real: the anchor and the formulation objective are both derived from the same 100 initial samples that later seed the optimization search. Nothing in the paper shows the selected formulation would be stable under resampling, so part of the reported gain could be a fit to that particular D_init. The conclusion even concedes anchor selection can be unreliable when samples miss hard regions—that's a load-bearing caveat. A held-out evaluation, e.g., select on one set of 100 points, evaluate on fresh starting sets, would resolve it.\n\nThere is also a minor inconsistency: the main text says a 300-evaluation budget for the antenna tasks, while the parameter sensitivity table's note says 200. That should be corrected.\n\nNone of these are fatal. The core idea holds up: different valid formulations of the same design intent do lead to very different search costs, and the paper demonstrates that convincingly. The baselines may be simple, but they are plausible, and the full formulations are provided so the comparison is checkable.\n\nBottom line: this deserves a serious referee—probably with a request for re-runs and resampling experiments—but it is a legitimate advance in a small area, and the authors did the work like researchers, not like a demo. I'd take it at a solid venue after a major revision.","headline":"A genuinely new anchor-state criterion for formulation search, but the efficiency claim rests on single runs and an in-sample selection that needs resampling tests.","tokens_in":27397,"tokens_out":2349,"would_cite":true,"duration_ms":26251,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A formulation that prioritizes rare, promising samples reaches requirement-satisfying designs with fewer expensive simulations.","keywords":["search-hardness-aware formulation","LLM-based problem formulation","expensive simulation-driven design","anchor state selection","satisfaction states","evolutionary formulation search","antenna design optimization","surrogate-assisted optimization"],"falsifier":"Run SHA-PF on a benchmark where the initial 100 evaluations are deliberately constrained to a region far from the known satisfying designs, so no rare promising state is observed. If SHA-PF then fails to reach the requirement within budget while a formulation that merely restates the thresholds succeeds, the claim that initial data reveal progress potential is refuted.","tokens_in":26504,"feed_emoji":"📡","tokens_out":4047,"duration_ms":47326,"temperature":0.7,"pith_summary":"The paper is trying to establish that the choice of problem formulation—not just the optimizer—controls how many expensive simulator calls are needed, and that a good formulation can be found automatically by reading search hardness out of the first batch of evaluated designs. It argues that initial space-filling evaluations contain partial paths to the target region: frequently observed satisfaction states are easy for an optimizer to reach anyway, whereas rare states mark hard but promising directions. The proposed method, SHA-PF, selects the rarest non-dominated state as an anchor and searches over LLM-generated formulations for the one that ranks anchor responses highest. In experiments on a hydrological calibration benchmark and five antenna design tasks, SHA-PF reaches requirement-satisfying designs in markedly fewer evaluations, succeeding on all five antenna tasks where most expert-based and direct-LLM baselines run out of budget.","feed_headline":"Rare-sample priority cuts costly simulations in antenna design","feed_subtitle":"A search-hardness-aware LLM formulation reaches all five antenna targets; expert and direct-LLM baselines mostly run out of budget.","key_machinery":"The anchor-state formulation objective (Eq. 7) is the load-bearing mechanism: select a non-dominated satisfaction state z_anc with minimal h(z) = occurrence_count(z) / (number_of_initial_points_dominated_by_z + epsilon), then score each candidate formulation by the proportion of pairs (anchor response, other response) in which the anchor ranks higher. This objective turns the abstract idea of 'search hardness' into a concrete fitness for formulation search, guiding the LLM-based evolutionary loop toward formulations that encourage the optimizer to follow hard but promising trajectories.","core_discovery":"The paper claims that, among valid formulations of the same design requirement, the ones that give higher priority to rarely observed satisfaction states with dominance support induce search trajectories that reach the requirement-satisfying region in fewer expensive simulator calls. To make this actionable, SHA-PF converts each candidate formulation into an executable objective-plus-constraints program, scores it by the fraction of anchor-vs-other pairs in which the anchor response receives higher priority, and evolves a population of formulations through LLM-based generation, repair, and four variation operators. The authors report that their discovered formulations require substantially f","pith_inferences":["If the finding generalizes, the same anchor-state scoring could be applied to any expensive black-box design setting with natural-language requirements, such as materials discovery or drug design, where the first batch of evaluations is already available.","The binary satisfaction state discards margin-to-threshold information; a graded version of the state could make anchor selection smoother, though it might also change which regions count as rare.","The paper's proposed iterative anchor discovery—using a few extra evaluations to update satisfaction states during formulation search—is a natural next test and could reduce the sensitivity to the initial sample placement.","The rarity-plus-dominance heuristic is one concrete proxy for progress potential; comparing it against surrogate-uncertainty-based or gradient-based anchors on the same benchmarks would clarify what exactly makes a state 'promising.'"],"forward_implications":["Formulation choice becomes a first-class lever for simulation budget: fixing the optimizer, different valid formulations of the same requirement produce materially different search costs.","The initial space-filling evaluations can be reused for formulation search before spending the main optimization budget, so the cost of finding a good formulation is mostly the initial samples themselves.","Formulations discovered by SHA-PF transfer across optimizers, suggesting the hardness signal is tied to the landscape rather than to one search algorithm.","Search hardness and design-intent alignment become separate, jointly necessary evaluation axes for automatic problem formulation in expensive design.","In the antenna benchmarks, the method succeeds where threshold-restating expert and direct-LLM formulations fail, indicating that graded, guidance-shaped formulations matter more than merely encoding the final acceptance criteria."],"fun_headline_variants":["LLM formulations that prioritize rare samples cut simulation costs","Search-hardness-aware LLM formulation cuts simulator calls","Antenna design: rare-sample priority in LLM formulations reduces simulations","LLM formulation with rare-sample priority needs fewer antenna simulations","Rare sample priority in LLM formulation lowers expensive simulator use"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that 100 space-filling evaluations expose the search-hardness structure of the whole design space; the paper itself concedes that when these samples are sparse or miss hard yet promising regions, anchor selection—and the formulation objective built on it—is misdirected.","fun_headline_variants_meta":{"raw":{"variants":["LLM formulations that prioritize rare samples cut simulation costs","Search-hardness-aware LLM formulation cuts simulator calls","Antenna design: rare-sample priority in LLM formulations reduces simulations","LLM formulation with rare-sample priority needs fewer antenna simulations","Rare sample priority in LLM formulation lowers expensive simulator use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001065,"raw_usage":{"total_tokens":4285,"prompt_tokens":711,"completion_tokens":3574,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":455,"completion_tokens_details":{"reasoning_tokens":3492}},"tokens_in":455,"tokens_out":3574,"duration_ms":23165,"temperature":1.0,"reasoning_tokens":3492,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T08:05:54.623195+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SHA-PF on a benchmark where the initial 100 evaluations are deliberately constrained to a region far from the known satisfying designs, so no rare promising state is observed. If SHA-PF then fails to reach the requirement within budget while a formulation that merely restates the thresholds succeeds, the claim that initial data reveal progress potential is refuted.","supporting_citations":[],"review_version":1}