Pith. sign in

REVIEW 2 major objections 6 minor 21 references

DCEDesignSA: A MATLAB-based Graphical User Interface for Discrete Choice Experiment Design Using Simulated Annealing

T0 review · 2 major / 6 minor · reviewed 2026-07-11 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An open MATLAB GUI uses simulated annealing to generate Bayesian D-optimal discrete choice designs and export them ready for Qualtrics.

desk verdict Solid software paper: open MATLAB GUI packaging prior SA work for Bayesian D-optimal DCEs, with Qualtrics export and order-balance support; efficiency claims rest on earlier papers, not new runs. read the letter →

arxiv 2607.04066 v1 pith:SLHHNBGU submitted 2026-07-05 stat.CO

Yinfu Liu , Yicheng Mao This is my paper · ORCID
classification stat.CO
keywords DiscreteChoiceExperimentSimulatedAnnealingMultinomiallogitmodelBayesianOptimalDesignMATLABD-optimalpartialprofile
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces DCEDesignSA, a free MATLAB toolbox with a graphical interface that builds Bayesian D-optimal designs for discrete choice experiments without requiring users to write code. It maximises the expected information about preference parameters by searching the design space with simulated annealing under a user-specified prior. The tool supports main effects plus interactions, partial-profile designs, no-choice (opt-out) alternatives, and presentation-order effects, then exports the finished design as Qualtrics-ready files. The authors position the package as filling a practical gap between commercial GUI tools and code-only open packages, giving non-programmers access to statistically efficient designs that account for prior uncertainty. A sympathetic reader cares because better experimental designs under realistic uncertainty improve the precision of preference estimates that inform marketing, health, and transport decisions.

What carries the argument

The Bayesian D-optimality criterion—the expected log-determinant of the multinomial-logit Fisher information matrix under a user prior—maximised by a simulated-annealing search whose exploration rule adapts to full-profile, partial-profile, and order-balanced designs.

What would settle it

Take a fixed set of design problems (with and without interactions or partial profiles), generate designs with DCEDesignSA under a stated time or cycle budget and with a coordinate-exchange package under the same prior, then compare the realised Bayesian D-errors and infinite-error rates.

Watch

Extended reading notes

Core claim

DCEDesignSA is an open-source MATLAB graphical toolbox that employs simulated annealing to construct Bayesian D-optimal designs for multinomial-logit discrete choice experiments. It lets users define attributes, model terms (including interactions), priors, and design settings through a four-panel interface, then returns designs that support full or partial profiles, opt-out alternatives, and balanced presentation order, with direct export to Qualtrics.

Load-bearing premise

The claim that simulated annealing yields superior statistical efficiency rests on earlier benchmark studies rather than new head-to-head runs performed inside this software paper.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. DCEDesignSA is presented as an open-source MATLAB toolbox (v1.0.0, MIT) with a GUI for constructing Bayesian D-optimal discrete choice experiment designs under the multinomial logit model. The package implements simulated annealing (Algorithm 1) to maximise the Bayesian D-criterion (Eq. 4) obtained by integrating the MNL Fisher information matrix (Eq. 3) over a user-specified prior via spherical-radial quadrature. Supported features include main effects and interactions, partial-profile designs, no-choice alternatives, presentation-order effects, and direct export to Qualtrics-compatible .txt and .csv. The manuscript supplies architecture (Fig. 1), a four-panel GUI workflow, a feature comparison with JMP, Ngene, idefix and choiceDes (Table 2), practical termination guidelines, and a laundry-detergent bottle walkthrough with reported D_B = 8.2926 and zero infinite-error rate.

Significance. If the software works as described, the contribution is practically useful: it lowers the barrier to Bayesian D-optimal DCE design for non-programmers, packages SA (an alternative to the CE/SWAP/MF algorithms used elsewhere), and closes the design-to-survey gap via Qualtrics export. Support for interactions, partial profiles, opt-out, and balanced profile order in one open GUI is a genuine gap relative to Table 2. Strengths include a public GitHub repository, explicit MNL/Bayesian formulae, a fully stated SA skeleton with reheating, and a concrete end-to-end example. The efficiency-superiority argument is delegated to prior peer-reviewed work rather than re-proved here, which is acceptable for a software paper provided the artifact itself is the primary claim.

major comments (2)
  1. Algorithm 1 (Section 3.2.2) invokes an unspecified “Exploration Rule” that “adapts to different design contexts, including full profile, partial profile, and balanced profile order design.” The candidate-move set is load-bearing for reproducibility of the method description: without stating which coordinates or profiles are swapped/exchanged under each design mode (and how balance constraints are enforced for order effects), readers cannot assess neighbourhood structure or verify that the GUI implements the SA procedure claimed in [11,13,14]. Please define the Exploration Rule formally (or in a short appendix) for each supported design type.
  2. Section 2 and the Impact section assert that SA yields superior statistical efficiency to CE designs from existing packages, citing only the authors’ prior papers [11,13,14]. For a software contribution this is not fatal, but the claim is motivational and currently unillustrated in this manuscript. A minimal head-to-head in the laundry-detergent example (e.g., Bayesian D-criterion and infinite-error rate under SA vs a CE/MF baseline for the same prior, S, J, and model) would let readers see the advantage without consulting three external papers and would confirm that the GUI implementation preserves the published gains.
minor comments (6)
  1. Section 3.2.2: the three termination criteria (adaptive / cycle / time) are recommended by design complexity, but “adaptive” is never defined operationally (what triggers stop? relative improvement? plateau length?). A one-sentence definition would match the clarity of the cycle and time options.
  2. Table 2: the comparison is restricted to D-optimal MNL designs, which is stated, but a footnote clarifying that Ngene and JMP support additional models/criteria (outside scope) would prevent over-reading the “No” cells as absolute feature absences.
  3. Metadata C5 lists “MATLAB R2025”; if the package also runs on earlier releases, stating the minimum tested version would help users. If R2025-only features of App Designer are required, say so explicitly.
  4. Section 4.1: the prior is described as “zero prior mean vector and an identity prior covariance matrix” plus an ASC mean of 1 for the opt-out. Confirm in the text whether the ASC is included in the identity covariance block or fixed, since that affects the dimension of β and of M(X,β).
  5. Presentation: several places in the supplied text show concatenated words (e.g., abstract opening, “DCEDesignSAisafreelyavailable”); if these appear in the PDF, re-export with proper spacing. Also standardise “D-optimal” hyphenation and “opt-out / no-choice” terminology throughout.
  6. Figure 5 caption mentions yellow highlighting for varying attributes; ensure the figure is colour-accessible or add a non-colour cue for print readers.

Circularity Check

1 steps flagged · score 2.0 of 10

Software paper is self-contained; only minor non-load-bearing self-citation for SA efficiency claims imported from prior Mao et al. work.

  1. self citation load bearing [Section 2 (Motivation and significance), paragraph on SA vs CE]
    "DCEDesignSA addresses this by employing the SA algorithm, which yields superior statistical efficiency compared to designs generated via the CE algorithm in existing packages, as demonstrated through a systematic benchmark across multiple design scenarios, including both full profile and partial profile settings [11, 13, 14]."

    The paper’s comparative efficiency claim for SA over CE is justified solely by three citations whose lead/co-author (Y. Mao) overlaps with the present authors; no independent replications or new head-to-head benchmarks appear in this manuscript. The claim is motivational for choosing SA, not definitional of the software artifact, so it is only mildly load-bearing.

full rationale

DCEDesignSA is a software-contribution paper whose central claim is the existence and usability of an open-source MATLAB GUI that packages Bayesian D-optimal DCE design generation via simulated annealing, with support for interactions, partial profiles, no-choice, order effects, and Qualtrics export. The MNL utility, Fisher information, and Bayesian D-criterion (Eqs. 1–4) are standard textbook constructions, not derived from fitted targets. Algorithm 1 is a conventional SA procedure with an exploration rule adapted to design type; it does not redefine its objective in terms of the reported designs. The sole circularity-adjacent element is the assertion that SA yields superior statistical efficiency to CE/SWAP/MF, which is justified only by citations [11,13,14] whose authors overlap with the present paper and is not re-demonstrated with fresh head-to-head runs inside this manuscript. That claim is motivational rather than definitional or forced by construction, and the artifact claim (GUI, architecture, export, supported features) stands independently of it. No self-definitional loop, fitted-input-as-prediction, uniqueness import, or renaming of a known result appears. Score 2 reflects one minor non-load-bearing self-citation chain; the derivation of the software contribution itself is self-contained.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper is a software contribution whose central claims rest on the classical MNL model, the Bayesian D-optimality integral, the SA meta-heuristic, and the authors’ earlier empirical benchmarks. No new physical entities or free parameters are fitted to data inside this manuscript; user-chosen priors and SA termination settings are configuration knobs, not fitted constants of a scientific claim.

free parameters (3)
  • SA temperature schedule and reheating rule
    Initial temperature T0 is set by random walk; cooling is T_k = T0/(k+1) with reheating after 1000 rejected moves (Algorithm 1). These are algorithmic hyper-parameters chosen by the implementers, not derived.
  • User-specified prior mean and covariance = zero mean, identity covariance (example)
    Bayesian D-criterion (Eq. 4) integrates over a user-supplied π(β); the illustrative example uses zero mean and identity covariance. Correctness of any generated design is conditional on this prior.
  • Termination criterion (adaptive / cycle / time)
    Users choose among adaptive, cycle (<10), or wall-clock time limits; the laundry example uses 600 s. Different choices can yield different designs.
assumptions (4)
  • domain assumption Respondent utilities follow the multinomial logit model with i.i.d. Type-I extreme-value errors (Eqs. 1–2).
    Standard random-utility assumption that yields closed-form choice probabilities and the information matrix (Eq. 3); invoked throughout Section 3.2.2.
  • domain assumption Bayesian D-optimality (integral of log-det information matrix over prior) is the appropriate design criterion.
    Adopted without derivation as the objective to maximize (Eq. 4); standard in the DCE optimal-design literature cited.
  • domain assumption Spherical-radial transformation sampling of Gotwalt et al. accurately approximates the Bayesian integral.
    Used for numerical evaluation of DB; package ships .pts/.wts files; justified by citation [20,21].
  • ad hoc to paper Simulated annealing with the stated exploration rule and reheating finds designs of higher statistical efficiency than coordinate exchange.
    Load-bearing performance claim imported from authors’ prior work [11,13,14] rather than re-proved here (Section 2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of DCEDesignSA: A MATLAB-based Graphical User Interface for Discrete Choice Experiment Design Using Simulated Annealing." pith.science (2026). https://pith.science/paper/SLHHNBGU

@misc{pith2026260704066,
  author       = {Pith},
  title        = {Pith review of: DCEDesignSA: A MATLAB-based Graphical User Interface for Discrete Choice Experiment Design Using Simulated Annealing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SLHHNBGU}},
  note         = {Machine review of arXiv:2607.04066}
}
read the original abstract

DCEDesignSA is a freely available MATLAB package for generating Bayesian D-optimal discrete choice experiment designs. It employs Simulated Annealing to efficiently search the design space and maximise the Bayesian D-optimality criterion under user-specified prior distributions. The toolbox features an interactive graphical user interface, enabling researchers without programming expertise to define experimental settings, generate optimal designs, and export survey-ready designs directly to Qualtrics. DCEDesignSA supports interaction terms in utility, no-choice alternatives, and presentation order effects.

Figures

Figures reproduced from arXiv: 2607.04066 by the authors.

Figure 1
Figure 1. DCEDesignSA: Software Architecture as the proportion of prior draws yielding an infinite D-error, where the D-error is the inverse of the Bayesian D-optimality criterion. The out￾put layer also supports direct export to a Qualtrics-compatible .txt file and a .csv spreadsheet. 3.2. Software functionalities 3.2.1. Experiment Design Specification As shown in [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. The DCEDesignSA graphical user interface showing the four-step configuration [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Laundry Detergent Bottle DCE Attributes Example [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Summary and level balance output for the laundry detergent bottle design [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: An example choice set displayed in the Qualtrics survey. Yellow highlighting [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 13 canonical work pages

  1. [1]

    J. Liu, B. Kassas, J. Lai, Investigating the Role of Political Messaging on Preferences for Local Food Products in the United States, Journal of Agricultural and Applied Economics 56 (3) (2024) 405–428.doi: 10.1017/aae.2024.22

    work page doi:10.1017/aae.2024.22 2024

  2. [2]

    M. D. Clark, D. Determann, S. Petrou, D. Moro, E. W. de Bekker- Grob, Discrete Choice Experiments in Health Economics: A Review 11 of the Literature, PharmacoEconomics 32 (9) (2014) 883–902.doi: 10.1007/s40273-014-0170-x

    work page doi:10.1007/s40273-014-0170-x 2014

  3. [3]

    Lancsar, J

    E. Lancsar, J. Louviere, Conducting Discrete Choice Experiments to Inform Healthcare Decision Making, PharmacoEconomics 26 (8) (2008) 661–677.doi:10.2165/00019053-200826080-00004

    work page doi:10.2165/00019053-200826080-00004 2008

  4. [4]

    M. C. J. Bliemer, J. M. Rose, Construction of experimental designs for mixed logit models allowing for correlation across choice observations, Transportation Research Part B: Methodological 44 (6) (2010) 720–734. doi:10.1016/j.trb.2009.12.004

    work page doi:10.1016/j.trb.2009.12.004 2010

  5. [5]

    K. E. Train, Discrete choice methods with simulation, Cambridge Uni- versity Press, Cambridge, 2003.doi:10.1017/CBO9780511753930

    work page doi:10.1017/cbo9780511753930 2003

  6. [6]

    Combining sources of preference data, in: D. A. Hensher, J. D. Swait, J. J. Louviere (Eds.), Stated Choice Methods: Analysis and Appli- cations, Cambridge University Press, Cambridge, 2000, pp. 227–251. doi:10.1017/CBO9780511753831.008

    work page doi:10.1017/cbo9780511753831.008

  7. [7]

    Huber, K

    J. Huber, K. Zwerina, The importance of utility balance in efficient choice designs, Journal of Marketing Research 33 (3) (1996) 307–317. doi:10.2307/3152127

    work page doi:10.2307/3152127 1996

  8. [8]

    Sandor, M

    Z. Sandor, M. Wedel, Designing conjoint choice experiments using man- agers’ prior beliefs, Journal of Marketing Research 38 (4) (2001) 430– 444.doi:10.1509/jmkr.38.4.430.18904

    work page doi:10.1509/jmkr.38.4.430.18904 2001

Show all 21 references
  1. [9]

    R. D. Cook, C. J. Nachtrheim, A comparison of algorithms for constructing exact D-optimal de- signs, Technometrics 22 (3) (1980) 315–324, _eprint: https://www.tandfonline.com/doi/pdf/10.1080/00401706.1980.10486162. doi:10.1080/00401706.1980.10486162

    1980 doi

  2. [10]

    R. K. Meyer, C. J. Nachtsheim, The coordinate-exchange algorithm for constructing exact optimal experimental designs, Technometrics 37 (1) (1995) 60–69, _eprint: https://doi.org/10.1080/00401706.1995.10485889.doi:10.1080/ 00401706.1995.10485889

    1995 doi

  3. [11]

    Y. Mao, R. Kessels, T. C. van der Zanden, Constructing bayesian op- timal designs for discrete choice experiments by simulated annealing, Journal of Choice Modelling 55 (2025) 100551.doi:10.1016/j.jocm. 2025.100551. 12

    2025 doi

  4. [12]

    Kessels, B

    R. Kessels, B. Jones, P. Goos, M. Vandebroek, An efficient algorithm for constructing bayesian optimal choice designs, Journal of Business & Economic Statistics 27 (2) (2009) 279–291. URLhttps://www.jstor.org/stable/27799083

    2009

  5. [13]

    Y. Mao, R. Kessels, Simulated Annealing for Model-Robust Partial Pro- file Choice Designs in Healthcare Preference Studies, arXiv:2604.04294 [stat.ME] (Apr. 2026).doi:10.48550/arXiv.2604.04294

    2026 doi arXiv

  6. [14]

    Y. Mao, R. Kessels, Optimal designs for mixture choice ex- periments by simulated annealing, Chemometrics and Intel- ligent Laboratory Systems 257 (2025) 105305.doi:https: //doi.org/10.1016/j.chemolab.2024.105305. URLhttps://www.sciencedirect.com/science/article/pii/ S0169743924002454

    2025 doi

  7. [15]

    URLhttps://www.jmp.com/en/support/jmp-documentation

    JMP documentation. URLhttps://www.jmp.com/en/support/jmp-documentation

  8. [16]

    URLhttps://www.choice-metrics.com

    ChoiceMetrics. URLhttps://www.choice-metrics.com

  9. [17]

    Traets, D

    F. Traets, D. G. Sanchez, M. Vandebroek, Generating optimal designs for discrete choice experiments in R: The idefix package, Journal of Statistical Software 96 (2020) 1–41.doi:10.18637/jss.v096.i03

    2020 doi

  10. [18]

    Horne, choiceDes: Design functions for choice studies (Jun

    J. Horne, choiceDes: Design functions for choice studies (Jun. 2018). URLhttps://cran.r-project.org/web/packages/choiceDes/ index.html

    2018

  11. [19]

    Y. Mao, R. Kessels, R. Mee, Beyond randomization: Design and analysis of discrete choice experiments in the presence of profile order effects within choice sets, Applied Stochastic Mod- els in Business and Industry 41 (5) (2025) e70043, _eprint: https://onlinelibrary.wiley.com...

    2025 doi

  12. [20]

    C. M. Gotwalt, B. A. Jones, D. M. Steinberg, Fast Computation of Designs Robust to Parameter Uncertainty for Nonlinear Settings, Tech- nometrics 51 (1) (2009) 88–95. URLhttps://www.jstor.org/stable/40586566

    2009

  13. [21]

    J. Yu, P. Goos, M. Vandebroek, Comparing different sampling schemes for approximating the integrals involved in the efficient design of stated 13 choice experiments, Transportation Research Part B: Methodological 44 (10) (2010) 1268–1289.doi:10.1016/j.trb.2010.02.005. 14

    2010 doi

Pith tools

Reviewed July 11, 2026 · model on record in the stance chip above.