{"id":"ba211054-f8f0-4363-be05-0c5d77092f7a","arxiv_id":"2607.16933","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Leave-one-out sample reuse, common random numbers, and bootstrap-based racing select the Bayesian-optimal experimental design from a finite set using up to ~100x fewer expensive model evaluations.","lead":"This paper builds an adaptive algorithm that picks the best experiment from a fixed menu when each simulation run is expensive. It reports cutting the number of simulation runs by one to two orders of magnitude while still selecting the design that long, expensive calculations would select.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ADE's reliability rests on the unproven §2.3 monotone-bias premise; a non-monotone bias counterexample would let ADE eliminate the true best before samples can correct it.","rationale":"The reader's weakest_assumption is also the most load-bearing concern I find. The paper is transparent and internally consistent, and the synthetic and pharmacokinetic examples provide real independent support: the closed-form EIG in §3.1 and N=10^6 references in §3.2/§3.3 make those benchmarks meaningful. However, the abstract and conclusion claim high reliability for expensive large-scale engineering applications, and the only expensive-model example (§3.4) has no independent ground truth: its reference EIG values are computed with the same estimator family at only N=10^3, where the two reference estimators disagree by up to a factor of 4. The bridge that would let the method extrapolate from cheap benchmarks to expensive models is precisely the §2.3 monotone-bias premise. That premise is empirical, single-example, and not guaranteed; if it fails, the bootstrap threshold gives confidence in a biased verdict and Algorithm 1 can irreversibly discard the true optimum before enough samples accumulate. This does not invalidate the method—it may well work in many problems—but it means the current evidence justifies conditional acceptance, not the unqualified generality claim. My proposed test directly probes the premise and the algorithm's behavior under a plausible violation, so it would settle whether the concern actually lands.","tokens_in":20059,"tokens_out":6685,"duration_ms":82711,"concrete_test":"Use the §3.1 closed-form setting to build two designs with known EIG such that the likelihood family makes the inner-sample variance of p(y|θ,ξ) larger for the lower-EIG design (e.g., Student-t noise with different degrees of freedom, or a mixture likelihood with a rare high-weight component). At N=8,16,32, estimate the ANMC bias over 10^4 independent replications and plot bias vs true EIG. Then run Algorithm 1 with the paper's defaults (Nnew=8, q=0.99, NC=3) on this pair for 1000 seeds. If the bias is not monotonically increasing or ADE eliminates the true optimal design in any run, the §2.3 premise fails and the headline reliability claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central reliability claim depends on the §2.3 premise that the bias of NMC/LRNMC/ANMC increases monotonically with true EIG, 'thereby amplifying differences between competing designs.' This premise is demonstrated on a single lognormal family (Figure 1) and justified only by a Jensen-gap heuristic; it is not a theorem. The bootstrap probability (16) is centered on the biased estimate, so a high value of \\hat P simply means the bootstrap replicates are consistently positive in the biased metric. If the bias is non-monotone or has the opposite sign for some high-EIG design, then at the small sample sizes used for elimination (Nnew=8) Algorithm 1 can confidently discard the true best design before later iterations add enough samples to correct the ranking; because eliminated designs are never revisited, the error is final. The expense regime is exactly where this matters: in §3.4 the two reference estimators differ by a factor of 4 at N=10^3, and the reported 100% ADE accuracy there is only agreement with the same biased estimator family, not with an independent ground truth.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an adaptive design elimination (ADE) algorithm for Bayesian optimal experimental design over finite design sets. The method combines nested Monte Carlo estimation with outer-sample reuse, a leave-one-out correction (LRNMC), Rao–Blackwellization (ANMC), common random numbers across designs, and bootstrap-based elimination of inferior designs. The central claim is that ADE with ANMC identifies the optimal design with high reliability while reducing the number of forward model evaluations by orders of magnitude relative to static design comparison (SDC). The method is demonstrated on four examples: an analytic lognormal test case, a pharmacokinetic model, a human lung model, and a viscoplastic finite-element model.","tokens_in":20221,"tokens_out":6775,"duration_ms":74531,"significance":"If the claims hold, the paper is a useful practical contribution. It packages several known variance-reduction ideas into an automatic elimination procedure with sensible defaults, and the analytic example provides a clean correctness check. The code is publicly available. However, the central reliability claim rests on an empirical monotone-bias assumption that is not theoretically justified, and the expensive-model validation uses references from the same biased estimator family. These issues must be addressed before the method can be recommended as 'high reliability' for general large-scale applications.","major_comments":[{"comment":"The reliability of ADE depends on the assertion that the bias of NMC/LRNMC/ANMC increases monotonically with the true EIG, thereby amplifying true differences between designs. This is demonstrated for one lognormal problem (Figure 1) and justified heuristically via Jensen's inequality. No theoretical analysis or broader evidence is provided. If the bias is not monotone, Eq. (16) can report high confidence in a wrong ordering, and Algorithm 1 may permanently eliminate the optimal design before later iterations correct the ranking, since eliminated designs are never revisited. This is a load-bearing assumption for the central reliability claim. The authors should either prove monotonicity under stated conditions, provide a robust elimination criterion that does not rely on it, or analyze the consequences when it fails.","section":"§2.3, Eq. (16), Algorithm 1"},{"comment":"The reference optimal design ξ9 is identified using RNMC and ANMC at N=10^3 samples. At this sample size the two estimators disagree dramatically (e.g., for ξ9, RNMC gives 6.662 and ANMC gives 26.185). This indicates that neither estimator has converged and that the 'true' ranking is uncertain. The reported 100% accuracy of ADE in this example is therefore only agreement with the same biased estimator family, not with an independent ground truth. Since this is the only example with an expensive forward model, the claim that the method is 'well-suited for large-scale engineering applications' is not fully supported. Please provide a more reliable reference (e.g., larger N for at least a subset of designs or an independent estimator) or qualify the conclusion.","section":"§3.4, Table 4 and Figure 9"},{"comment":"The elimination rule has no formal error control. The bootstrap probability (16) is not a calibrated p-value, and the thresholds q=0.99 and NC=3 are heuristic. At early iterations with N=8, the bootstrap is based on very few samples and can be highly discrete; the paper does not analyze the probability of incorrectly eliminating the optimal design. The hyperparameter study in Appendix C is useful but empirical. For a method whose central promise is reliability, the authors should provide a finite-sample or asymptotic analysis of the elimination decision, or at least a conservative correction to the bootstrap threshold.","section":"§2.4, Algorithm 1"}],"minor_comments":[{"comment":"The line 'Identify reference design: k* = argmax_k P^(k)' is ambiguous because P^(k) is defined as a vector of per-sample PMI estimates. It should read 'argmax_k (1/|Θ|) Σ_i P_i^(k)' or otherwise clarify that the mean PMI is used.","section":"Algorithm 1"},{"comment":"The claim of 'two orders of magnitude' reduction in model evaluations would be easier to verify if the figure included explicit axis markers or tabulated cost values at the 100% accuracy points.","section":"§3.1, Figure 2"},{"comment":"Given the large discrepancy between RNMC and ANMC reference values, reporting standard errors or confidence intervals for the N=10^3 estimates would help the reader assess convergence.","section":"§3.4, Table 4"},{"comment":"The statement that 'runs are not fully independent' because samples are drawn from a precomputed set could affect the reported accuracy values. Please quantify the dependence or sample fresh inputs for at least a subset of runs.","section":"§3.3"},{"comment":"The term 'accelerated' in ANMC is not explicitly justified; consider briefly noting that the acceleration comes from avoiding repeated model evaluations via Rao–Blackwellization.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-written and the experimental study is broad, but the main risk is overclaiming reliability based on an unproven bias property and self-referential validation. I believe the paper can be made acceptable with additional theoretical or numerical support, but as it stands the central claim is not fully established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll keep this short. The genuinely new pieces are the leave-one-out RNMC estimator and the Rao-Blackwellized ANMC variant. The first removes the negative bias that sample reuse introduces while keeping most of the reuse benefit; the second averages over multiple noise samples per model evaluation, which is nearly free and cuts variance without extra forward solves. Both are well motivated and clearly presented. The adaptive elimination with bootstrap and common random numbers is a reasonable assembly of known machinery, and the synthetic benchmark with a near-tie between the top two designs is a hard, honest test that the method passes. Equations are clean, appendices are correct, and the numerical work uses 1000 runs per configuration. The comparison with static design choice is fair because both use the same estimators and the same evaluation counting. Code is public.\n\nThe main soft spot is the 'facilitating bias' premise in §2.3. They argue that the bias of NMC/LRNMC/ANMC increases with true EIG, so it amplifies the difference between designs and makes elimination reliable. That is shown empirically on one lognormal test problem and justified with a Jensen-gap heuristic. The bootstrap probability (16) is centered on the biased estimate, so a high value means the biased estimator is confident in itself. If the monotonicity fails in some application, ADE can eliminate the true best before larger N corrects the ranking, and eliminated designs are never revisited. With NC=3 and steadily growing N the risk is smaller than the worst-case sketch, but this should be treated as a condition, not a general guarantee.\n\nRelatedly, in §3.4 the reference EIG values are computed with N=10^3, and RNMC and ANMC disagree by a factor of about 4 for the top design (6.66 vs 26.19). The 100% accuracy in that section is mostly self-consistency with the same biased estimator family, not agreement with an independent ground truth. The synthetic example is the only one with an analytic EIG, and there the method performs well.\n\nThese are not errors in the estimator equations. They are limitations on the generality of the reliability claim. I'd like an external benchmark, an N_epsilon sensitivity study, and error bars on the reference EIG values before trusting the two-orders-of-magnitude claim in general. But the LRNMC and ANMC ideas are worth citing, and the paper deserves serious refereeing. The referee should press on the monotone-bias assumption and the §3.4 reference inconsistency.","headline":"Solid assembly of known tricks with two genuinely useful estimator variants (LRNMC, ANMC); the reliability claim leans on an unproven monotone-bias assumption, but the paper deserves serious refereeing.","tokens_in":20877,"tokens_out":5272,"would_cite":true,"duration_ms":55293,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62K05","62F15","65C05"],"pacs":[],"model":"deepseek-v4-flash","headline":"An adaptive elimination algorithm for Bayesian optimal experimental design identifies the best experiment in a finite set with orders of magnitude fewer expensive forward-model evaluations than standard nested Monte Carlo, without sacrifici","keywords":["Bayesian optimal experimental design","finite design sets","nested Monte Carlo","adaptive design elimination","Rao-Blackwellization","common random numbers","bootstrap confidence","expected information gain"],"falsifier":"Repeat the bias experiment of Figure 1 with a problem where the likelihood terms are less variable for higher-EIG designs, so the Jensen gap does not grow with EIG; if the ANMC bias is then flat or decreasing in true EIG, run the ADE algorithm on a known finite design set and check whether the true best design is ever eliminated with bootstrap probability above 0.99 and never recovered by later iterations.","tokens_in":19759,"feed_emoji":"🎯","tokens_out":8020,"duration_ms":74850,"temperature":0.7,"pith_summary":"Bayesian optimal experimental design usually forces a choice between expensive nested Monte Carlo estimates and biased local approximations. This paper argues that when the candidate designs form a finite set, the goal should not be to estimate information gain accurately but to compare designs reliably. It introduces an adaptive elimination algorithm that reuses parameter samples across the inner and outer Monte Carlo loops, excludes the current sample to keep the bias benign, averages over noise via Rao-Blackwellization, and uses common random numbers plus bootstrap confidence to delete inferior designs early. The result is a reduction in required model evaluations of about two orders of magnitude relative to a fixed-budget reuse baseline, while the optimal design is identified in essentially all test runs. If this holds for a target application, BOED becomes practical for expensive finite-element and multiphysics simulators.","feed_headline":"Adaptive search finds best experiment with 100x fewer runs","feed_subtitle":"A bootstrap-based elimination scheme compares sensor placements or loading protocols with hundreds of model runs, not millions.","key_machinery":"The central object is the ANMC estimator: it uses a single prior sample set for both the outer expectation and the inner marginal-likelihood estimate, omits the current index in the inner sum, and averages pointwise mutual information over multiple noise samples per outer sample. This makes each design's EIG estimate cheap, low-variance, and free of the negative sample-reuse bias. On top of it sits the adaptive design elimination loop, which shares one sample set across all designs (common random numbers), computes bootstrap probabilities of pairwise superiority, and eliminates a design only after the probability that it is worse than the current best exceeds 0.99 for three consecutive itera","core_discovery":"The paper's central claim is that the best experiment from a finite, unordered set can be identified without accurately estimating every expected information gain. Instead, the algorithm pairs an Accelerated Nested Monte Carlo (ANMC) estimator—which reuses outer-loop parameter samples in the inner marginal-likelihood sum, leaves out the current sample to remove a negative reuse bias, and Rao-Blackwellizes over noise draws—with a bootstrap-based elimination loop that deletes designs only after three consecutive confident inferiority verdicts. Because common random numbers across designs make the pairwise information-gain differences low-variance, and because the ANMC bias tends to grow with t","pith_inferences":["The paper leaves implicit that the efficiency gains are empirical rather than proven; a natural extension is a sample-complexity bound for the elimination rule under the monotone-bias premise, which would tell practitioners when the two-order-of-magnitude saving is guaranteed.","The bootstrap elimination threshold and persist count behave like a fixed-confidence racing procedure; connecting them to an error budget could turn the method into a procedure with a formal reliability guarantee.","If a target application violates the monotone-bias premise, a safer variant could combine ADE with a debiased or multilevel estimator, accepting more evaluations in exchange for a guarantee that the true best design is never eliminated early.","The method currently assumes a deterministic forward model with stochastic noise; extending the reuse and leave-out logic to stochastic simulators would open the same efficiency gains to a wider class of applications."],"forward_implications":["BOED over finite design sets becomes feasible for expensive simulators: the ADE+ANMC combination identifies the optimal design with on the order of 10^2 to 10^3 model evaluations in the tested examples, where fixed-budget nested Monte Carlo would need 10^5 to 10^6.","The leave-out correction is a necessary ingredient: reusing the current outer sample in the inner loop adds a negative bias that compresses EIG differences and can make a worse design look competitive.","The algorithm is estimator-agnostic and black-box friendly: it requires only likelihood evaluations, not gradients or Gaussian approximations, and it works with non-Gaussian observation models.","The default hyperparameters (8 new samples per iteration, 0.99 probability threshold, 3 consecutive eliminations, 10^5 bootstrap samples) transfer across the four examples, so a practitioner can run ADE without tuning."],"fun_headline_variants":["Adaptive elimination finds best experiment in 100x fewer runs","Bootstrap design search: 100x less compute, same reliability","Pick top sensor layout with 1% of the simulation budget","Design elimination via bootstrap speeds up search 100x","Skip the million-run brute force: 100x faster design choice"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The algorithm's reliability hinges on the empirical premise that the ANMC estimator's bias increases monotonically with the true expected information gain; if this 'facilitating bias' fails, the bootstrap confidence can certify a biased comparison and eliminate the true best design before more samples correct the ranking.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive elimination finds best experiment in 100x fewer runs","Bootstrap design search: 100x less compute, same reliability","Pick top sensor layout with 1% of the simulation budget","Design elimination via bootstrap speeds up search 100x","Skip the million-run brute force: 100x faster design choice"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000916,"raw_usage":{"total_tokens":3766,"prompt_tokens":739,"completion_tokens":3027,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":2941}},"tokens_in":483,"tokens_out":3027,"duration_ms":21924,"temperature":1.0,"reasoning_tokens":2941,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T19:31:30.565684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Repeat the bias experiment of Figure 1 with a problem where the likelihood terms are less variable for higher-EIG designs, so the Jensen gap does not grow with EIG; if the ANMC bias is then flat or decreasing in true EIG, run the ADE algorithm on a known finite design set and check whether the true best design is ever eliminated with bootstrap probability above 0.99 and never recovered by later iterations.","supporting_citations":[],"review_version":1}