pith. sign in

arxiv: 2407.16212 · v2 · submitted 2024-07-23 · 📊 stat.ME · cs.NA· math.NA· stat.CO

Optimal experimental design: Formulations and computations

Xun Huan , Jayanth Jagalur , Youssef Marzouk This is my paper

Pith reviewed 2026-05-23 23:11 UTC · model grok-4.3

classification 📊 stat.ME cs.NAmath.NAstat.CO
keywords optimal experimental designBayesian designdecision-theoretic designsequential designinformation criterianonlinear modelscomputational design methods
0
0 comments X

The pith

Bayesian and decision-theoretic methods give optimal experimental design a flexible framework suited to nonlinear and non-Gaussian models.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The survey traces optimal experimental design from its classical roots to modern use with complex statistical models. It presents the Bayesian decision-theoretic view as a unifying way to encode experimental goals through expected utility or information measures. This view supports design criteria that remain well-defined even when models are nonlinear, high-dimensional, or yield non-Gaussian posteriors. The paper then examines practical computation of those criteria and the search for good designs, including sequential policies that adapt as data arrive.

Core claim

Optimal experimental design problems can be stated by choosing among criteria that quantify the value of an experiment, with the Bayesian and decision-theoretic formulation providing a consistent way to incorporate prior knowledge and to handle nonlinear or non-Gaussian models; values of these criteria can be estimated or bounded by a range of Monte Carlo and deterministic methods, designs can be optimized over discrete or continuous spaces, and sequential policies can be constructed to coordinate an entire sequence of experiments rather than choosing each one myopically.

What carries the argument

The Bayesian decision-theoretic formulation of OED, which selects a design by maximizing the expected value of a utility function computed from the posterior distribution that would result after the experiment is performed.

If this is right

  • Information-based criteria become directly usable for models whose likelihoods are nonlinear or produce non-Gaussian posteriors.
  • Monte Carlo and bounding techniques can evaluate design criteria even when per-sample simulation cost is high or the model is implicit.
  • Both combinatorial selection of observation locations and continuous parameterization of designs admit practical optimization algorithms.
  • Sequential policies can be built that plan multiple future experiments jointly rather than one at a time.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same decision-theoretic setup could be used to compare the value of experiments that differ in cost or in the type of data they return.
  • Open computational challenges listed in the survey point to a need for tighter integration between design optimization and modern gradient-based or simulation-based inference tools.
  • Sequential non-myopic policies naturally extend to settings where the experimental budget itself is uncertain or must be allocated across competing scientific questions.

Load-bearing premise

The methods and literature surveyed here give a representative picture of current work on OED for complex models.

What would settle it

A concrete nonlinear non-Gaussian model in which every information-based Bayesian criterion produces designs that yield worse posterior predictions than a classical linear criterion would produce on the same problem.

Figures

Figures reproduced from arXiv: 2407.16212 by Jayanth Jagalur, Xun Huan, Youssef Marzouk.

Figure 2.1
Figure 2.1. Figure 2.1: Optimal sensor placement in a time-dependent advection-diffusion [PITH_FULL_IMAGE:figures/full_fig_p020_2_1.png] view at source ↗
Figure 3.1
Figure 3.1. Figure 3.1: Estimated EIG as a function of a scalar design parameter [PITH_FULL_IMAGE:figures/full_fig_p040_3_1.png] view at source ↗
Figure 3.2
Figure 3.2. Figure 3.2: Variational upper (orange) and lower (blue) bounds on the EIG in [PITH_FULL_IMAGE:figures/full_fig_p045_3_2.png] view at source ↗
Figure 5.1
Figure 5.1. Figure 5.1: In the MDP progression of sequential experimental design, we start [PITH_FULL_IMAGE:figures/full_fig_p080_5_1.png] view at source ↗
Figure 5.2
Figure 5.2. Figure 5.2: A policy is a mapping from state to design. In this DNN representation [PITH_FULL_IMAGE:figures/full_fig_p087_5_2.png] view at source ↗
Figure 5.3
Figure 5.3. Figure 5.3: Expected utility contours for the linear-Gaussian benchmark with the [PITH_FULL_IMAGE:figures/full_fig_p089_5_3.png] view at source ↗
Figure 5.4
Figure 5.4. Figure 5.4: An example time-evolution of a convection-diffusion field. The con [PITH_FULL_IMAGE:figures/full_fig_p091_5_4.png] view at source ↗
Figure 5.5
Figure 5.5. Figure 5.5: Sequence of marginal posterior densities (unknown source locations [PITH_FULL_IMAGE:figures/full_fig_p091_5_5.png] view at source ↗
Figure 5.6
Figure 5.6. Figure 5.6: Histograms of total rewards from 104 test trajectories generated using batch, greedy, and PG-sOED policies, with respective expected total reward values (indicated by vertical lines) 𝑈(𝜋 ∗ batch) ≈ 2.856, 𝑈(𝜋 ∗ greedy) ≈ 3.057, and 𝑈(𝜋PG) ≈ 3.435. Figure adapted from Shen and Huan (2023). Sutton and Barto (2018, Chapter 13). The expected utility 𝑈(𝜋) in DAD is the total EIG in the parameters Θ from prior… view at source ↗
Figure 5.7
Figure 5.7. Figure 5.7: Expected utility lower bounds achieved by various sequential exper [PITH_FULL_IMAGE:figures/full_fig_p098_5_7.png] view at source ↗
read the original abstract

Questions of `how best to acquire data' are essential to modeling and prediction in the natural and social sciences, engineering applications, and beyond. Optimal experimental design (OED) formalizes these questions and creates computational methods to answer them. This article presents a systematic survey of modern OED, from its foundations in classical design theory to current research involving OED for complex models. We begin by reviewing criteria used to formulate an OED problem and thus to encode the goal of performing an experiment. We emphasize the flexibility of the Bayesian and decision-theoretic approach, which encompasses information-based criteria that are well-suited to nonlinear and non-Gaussian statistical models. We then discuss methods for estimating or bounding the values of these design criteria; this endeavor can be quite challenging due to strong nonlinearities, high parameter dimension, large per-sample costs, or settings where the model is implicit. A complementary set of computational issues involves optimization methods used to find a design; we discuss such methods in the discrete (combinatorial) setting of observation selection and in settings where an exact design can be continuously parameterized. Finally we present emerging methods for sequential OED that build non-myopic design policies, rather than explicit designs; these methods naturally adapt to the outcomes of past experiments in proposing new experiments, while seeking coordination among all experiments to be performed. Throughout, we highlight important open questions and challenges.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

0 major / 2 minor

Summary. The paper presents a systematic survey of modern optimal experimental design (OED), beginning with foundations in classical design theory and extending to current research on OED for complex models. It reviews criteria for formulating OED problems (emphasizing the flexibility of Bayesian and decision-theoretic approaches that encompass information-based criteria suitable for nonlinear and non-Gaussian models), methods for estimating or bounding these criteria under challenges like high dimensionality or implicit models, optimization techniques for discrete observation selection and continuously parameterized designs, and emerging sequential OED methods that construct non-myopic policies adapting to past outcomes while coordinating across experiments. Open questions and challenges are highlighted throughout.

Significance. As a survey, the manuscript organizes and synthesizes the OED literature with a focus on computational aspects and the advantages of Bayesian formulations for complex settings; this synthesis is valuable for researchers in statistics, engineering, and applied sciences seeking an entry point or overview of methods for data acquisition. The paper does not derive new results but provides a structured review that could aid in identifying computational challenges and open problems in the field.

minor comments (2)
  1. [Abstract] Abstract: the phrase 'systematic survey' would benefit from a brief statement of literature selection criteria or time frame covered to clarify scope for readers.
  2. The discussion of sequential OED could include a short table comparing myopic vs. non-myopic approaches to improve readability.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive review and recommendation to accept the manuscript. The referee's summary correctly reflects the paper's scope as a survey of optimal experimental design formulations, computational methods, and open challenges.

Circularity Check

0 steps flagged

Survey paper reviews established literature; no new derivations or self-referential predictions

full rationale

This is a systematic survey of OED from classical foundations to modern computational methods for complex models. It reviews criteria, estimation techniques, optimization approaches, and sequential design without presenting original derivations, fitted parameters, or predictions that reduce to quantities defined within the paper itself. The highlighted flexibility of Bayesian/decision-theoretic OED (encompassing information-based criteria for nonlinear/non-Gaussian models) is described as an established property of the framework, not a novel result derived here. No load-bearing steps rely on self-citation chains, ansatzes smuggled via prior work, or renaming of known results as new unifications. The paper's scope is explicitly that of a review, making it self-contained against external benchmarks with no internal circularity.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

As a survey paper the work introduces no new free parameters, axioms, or invented entities; it relies on standard statistical design theory and Bayesian methods from the reviewed literature.

axioms (1)
  • standard math Standard assumptions of statistical modeling and design theory underpin the reviewed criteria and methods
    The survey builds directly on classical design theory and Bayesian decision-theoretic frameworks described in the abstract.

pith-pipeline@v0.9.0 · 5782 in / 1046 out tokens · 22990 ms · 2026-05-23T23:11:18.033819+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

25 extracted references · 25 canonical work pages · 2 internal anchors

  1. [1]

    P. K. Agarwal, S. Har-Peled and K. R. Varadarajan (2005), Geometric approximation via coresets, Combinatorial and Computational Geometry 52, 1–30. R. Aggarwal, M. J. Demkowicz and Y. M. Marzouk (2016), Information-driven experi- mental design in materials science, in Information Science for Materials Discovery and Design (T. Lookman, F. Alexander and K. R...

    work page doi:10.1007/s10208-023-09630-x 2005

  2. [2]

    80 of Proceedings of Machine Learning Research, PMLR, pp

    , Vol. 80 of Proceedings of Machine Learning Research, PMLR, pp. 531–540. A. Ben-Tal and A. Nemirovski (2001),Lectures on Modern Convex Optimization, SIAM. P. Benner, S. Gugercin and K. Willcox (2015), A Survey of projection-based model reduc- tion methods for parametric dynamical systems, SIAM Review 57(4), 483–531. J. O. Berger (1985), Statistical Decis...

    work page 2001

  3. [3]

    Chaudhuri, S

    (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu and S. Sabato, eds), Vol. 162 of Proceedings of Machine Learning Research, PMLR, pp. 2107–2128. 110 X. Huan, J. Jagalur and Y . Marzouk J. R. Blum (1954), Multidimensional stochastic approximation methods, The Annals of Mathematical Statistics 25(4), 737–744. N. Bochkina (2019), Bernstein–von Mises...

    work page 1954

  4. [4]

    Bogunovic, J

    I. Bogunovic, J. Zhao and V. Cevher (2018), Robust maximization of non-submodular objectives, in Proceedings of the Twenty-First International Conference on Artificial Intelligence and Statistics (A. Storkey and F. Perez-Cruz, eds), Vol. 84 of Proceedings of Machine Learning Research, PMLR, pp. 890–899. C. Borges and G. Biros (2018), Reconstruction of a c...

    work page arXiv 2018

  5. [5]

    Cui and X

    T. Cui and X. T. Tong (2022), A unified performance analysis of likelihood-informed subspace methods, Bernoulli 28(4), 2788–2815. T. Cui, S. Dolgov and O. Zahm (2023), Scalable conditional deep inverse Rosenblatt transports using tensor trains and gradient-based dimension reduction,Journal of Com- putational Physics 485, 112103. T. Cui, K. J. H. Law and Y...

    work page arXiv 2022

  6. [6]

    Eskenazis and Y

    A. Eskenazis and Y. Shenfeld (2024), Intrinsic dimensional functional inequalities on model spaces, Journal of Functional Analysis 286(7), 110338. S. Eswar, V. Rao and A. K. Saibaba (2024), Bayesian D-optimal experimental designs via column subset selection. Available at arXiv:2402.16000. K. Fan (1967), Subadditive functions on a distributive lattice and ...

    work page arXiv 2024

  7. [7]

    I. Ford, D. M. Titterington and C. P. Kitsos (1989), Recent advances in nonlinear experi- mental design, Technometrics 31(1), 49–60. A. Foster, D. R. Ivanova, I. Malik and T. Rainforth (2021), Deep adaptive design: Amort- izing sequential Bayesian experimental design, inProceedings of the 38th International Conference on Machine Learning (ICML 2021)(M. Me...

    work page 1989

  8. [8]

    Gneiting and A

    T. Gneiting and A. E. Raftery (2007), Strictly proper scoring rules, prediction, and estim- ation, Journal of the American Statistical Association 102(477), 359–378. J. Go and T. Isaac (2022), Robust expected information gain for optimal Bayesian ex- perimental design using ambiguity sets, in Proceedings of the 38th Conference on Uncertainty in Artificial...

    work page 2007

  9. [9]

    Gr¨ unwald and T

    P. Gr¨ unwald and T. van Ommen (2017), Inconsistency of Bayesian inference for misspe- cified linear models, and a proposal for repairing it,Bayesian Analysis12(4), 1069–1103. G. G¨ urkan, A. Y.¨Ozge and S. M. Robinson (1994), Sample-path optimization in simulation, in Proceedings of the 1994 Winter Simulation Conference (WSC ’94)(J. D. Tew, M. S. Manivan...

    work page 2017

  10. [10]

    Hairer, A

    M. Hairer, A. M. Stuart and J. Voss (2011), Signal processing problems on function space: Bayesian formulation, stochastic PDEs and effective MCMC methods, in The Oxford Handbook of Nonlinear Filtering (D. Crisan and B. Rozovskii, eds), Oxford University Press, pp. 833–873. O. Harari and D. M. Steinberg (2014), Optimal designs for Gaussian process models ...

    work page 2011

  11. [11]

    Sequential Bayesian optimal experimental design via approximate dynamic programming

    , Vol. 32 of Proceedings of Machine Learning Research, PMLR, pp. 739–747. D. S. Hochba (1997), Approximation algorithms for NP-hard problems, ACM SIGACT News 28(2), 40–52. X. Huan (2015), Numerical approaches for sequential Bayesian optimal experimental design, PhD thesis, Massachusetts Institute of Technology. X. Huan and Y. M. Marzouk (2013), Simulation...

    work page internal anchor Pith review Pith/arXiv arXiv 1997

  12. [12]

    Krause and D

    Optimal experimental design 119 A. Krause and D. Golovin (2014), Submodular function maximization, in Tractability, Practical Approaches to Hard Problems (B. Lucas, H. Youssef and K. Pushmeet, eds), Cambridge University Press, pp. 71–104. A. Krause, A. Singh and C. Guestrin (2008), Near-optimal sensor placements in Gaussian processes: Theory, efficient al...

    work page 2014

  13. [13]

    , ACM, pp. 826–839. L. C. Lau and H. Zhou (2022), A local search framework for experimental design, SIAM Journal on Computing 51(4), 900–951. L. Le Cam (1964), Sufficiency and approximate sufficiency, The Annals of Mathematical Statistics 35(4), 1419–1455. E. L. Lehmann and G. Casella (1998), Theory of Point Estimation , Springer Texts in Statistics, Spri...

    work page 2022

  14. [14]

    N. A. Letizia, N. Novello and A. M. Tonello (2023), Variational𝑓 -divergence and derange- ments for discriminative mutual information estimation. Available at arXiv:2305.20025. D. D. Lewis (1995), A sequential algorithm for training text classifiers: Corrigendum and additional data, SIGIR Forum 29(2), 13–19. F. Li, R. Baptista and Y. Marzouk (2024 a), Exp...

    work page arXiv 2023

  15. [15]

    J. A. Melendez, R. J. Furnstahl, H. W. Grießhammer, J. A. McGovern, D. R. Phillips and M. T. Pratola (2021), Designing optimal experiments: An application to proton Compton scattering, The European Physical Journal A 57, 1–24. Optimal experimental design 121 P. Mertikopoulos, N. Hallak, A. Kavis and V. Cevher (2020), On the almost sure conver- gence of st...

    work page arXiv 2021

  16. [16]

    R. T. Rockafellar and J. O. Royset (2015), Measures of residual risk with connections to re- gression, risk tracking, surrogate models, and ambiguity,SIAM Journal on Optimization 25(2), 1179–1208. 124 X. Huan, J. Jagalur and Y . Marzouk R. T. Rockafellar and S. Uryasev (2002), Conditional value-at-risk for general loss distri- butions, Journal of Banking ...

    work page arXiv 2015

  17. [17]

    Ruppert (1988), Efficient estimations from a slowly convergent Robbins-Monro pro- cess, Technical report, Cornell University

    D. Ruppert (1988), Efficient estimations from a slowly convergent Robbins-Monro pro- cess, Technical report, Cornell University. Available at http://ecommons.cornell.edu/ bitstream/handle/1813/8664/TR000781.pdf?sequence=1. L. Ruthotto, J. Chung and M. Chung (2018), Optimal experimental design for inverse problems with state constraints, SIAM Journal on Sc...

    work page arXiv 1988

  18. [18]

    Sun and W

    N.-Z. Sun and W. W. Yeh (2007), Development of objective-oriented groundwater models:

    work page 2007

  19. [19]

    Robust experimental design, Water Resources Research43(2), 1–14. R. S. Sutton and A. G. Barto (2018), Reinforcement Leaning, second edition, MIT Press. Optimal experimental design 127 R. S. Sutton, D. McAllester, S. P. Singh and Y. Mansour (1999), Policy gradient methods for reinforcement learning with function approximation, inAdvances in Neural Informat...

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  20. [20]

    , ACM, pp. 783–792. A. Wald (1943), On the efficient design of statistical investigations, The Annals of Math- ematical Statistics 14(2), 134–140. S. G. Walker (2016), Bayesian information in an experiment and the Fisher information distance, Statistics & Probability Letters 112, 5–9. J. Wang, S. C. Clark, E. Liu and P. I. Frazier (2020), Parallel Bayesia...

    work page arXiv 1943

  21. [21]

    K. Wu, P. Chen and O. Ghattas (2023 a), An offline-online decomposition method for efficient linear Bayesian goal-oriented optimal experimental design: Application to optimal sensor placement, SIAM Journal on Scientific Computing 45(1), B57–B77. K. Wu, T. O’Leary-Roseberry, P. Chen and O. Ghattas (2023 b), Large-scale Bayesian optimal experimental design ...

    work page 2023

  22. [22]

    H. P. Wynn (1972), Results in the theory and construction of D-optimum experimental designs, Journal of the Royal Statistical Society: Series B (Methodological) 34(2), 133–147. H. P. Wynn (1984), Jack Kiefer’s contributions to experimental design, The Annals of Statistics 12(2), 416–423. Z. Xu and Q. Liao (2020), Gaussian process based expected informatio...

    work page 1972

  23. [23]

    Yates (1933), The principles of orthogonality and confounding in replicated experiments, The Journal of Agricultural Science 23(1), 108–145

    F. Yates (1933), The principles of orthogonality and confounding in replicated experiments, The Journal of Agricultural Science 23(1), 108–145. F. Yates (1937), The design and analysis of factorial experiments. Technical Communica- tion no. 35, Imperial Bureau of Soil Science. F. Yates (1940), Lattice squares,The Journal of Agricultural Science 30(4), 672...

    work page 1933

  24. [24]

    Zhan and H

    D. Zhan and H. Xing (2020), Expected improvement for expensive optimization: A review, Journal of Global Optimization 78(3), 507–544. J. Zhang, S. Bi and G. Zhang (2021), A scalable gradient-free method for Bayesian experimental design with implicit models, in Proceedings of the 24th International Conference on Artificial Intelligence and Statistics, Vol....

    work page arXiv 2020

  25. [25]

    Zong (1999), Sphere Packings, Springer

    C. Zong (1999), Sphere Packings, Springer

    work page 1999