Pith. sign in

REVIEW 5 major objections 4 minor 35 references

Improving Policy-Oriented Agent-Based Modeling with History Matching: A Case Study

T0 review · 5 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read History matching cut the Covasim calibration problem from more than 100,000 simulator runs to 5,300 while retaining fits to empirical COVID-19 data across random seeds.

desk verdict A credible, well-documented case study showing history matching + emulation + ABC can cut Covasim calibration cost by ~20x, but the paper overclaims 'out-of-sample' fit because the only evidence is visual and the same data are reused. read the letter →

arxiv 2501.00616 v1 pith:WHLZDRVW submitted 2024-12-31 stat.AP stat.CO

David O'Gara , Cliff C. Kerr , Daniel J. Klein , Mickaël Binois , Roman Garnett , Ross A. Hammond This is my paper · ORCID
classification stat.APstat.CO MSC 62F1562P10
keywords historymatchingagent-basedmodelCovasimapproximateBayesiancomputationGaussianprocessemulationcalibrationCOVID-19uncertaintyquantification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that history matching — a Bayesian technique that uses fast surrogate models ('emulators') to rule out parameter values unlikely to match observed data — can make calibration of expensive agent-based models practical for time-sensitive policy work. In a case study with Covasim, a widely used COVID-19 model, four rounds of history matching ruled out more than 99 percent of the candidate parameter space (from 2,560,000 grid points to 21,114), and a follow-up approximate Bayesian computation (ABC) posterior, built from just 5,300 simulator runs, matched the empirical time-series of diagnoses and deaths across random seeds. The original calibration of the same model required more than 100,000 runs, about 35 days of computing. If the claim holds, it means high-resolution mechanistic models can be calibrated quickly enough to inform real-time outbreak response, and can carry honest uncertainty about which parameter settings are consistent with the data.

What carries the argument

The load-bearing object is the implausibility measure $I(\theta) = |Y - \hat{\mu}(\theta)| / \sqrt{\hat{\sigma}_{\text{em}}^2(\theta) + \sigma_{\text{model disc}}^2 + \sigma_{\text{obs}}^2}$, maximized across the output time-series; parameterizations with maximum implausibility above a cutoff (3, then 2.7, then 2.5) are removed. The emulators are heteroskedastic Gaussian processes (hetGP) estimated with the 'hetGPy' package, which exploits the Woodbury matrix identity so that training cost scales with the number of unique design locations $n$ rather than the total number of runs $N$, making 20-plus replicates per location affordable and letting the noise vary across input space. Sequential waves re-fit emulators on the remaining NROY region with denser output timesteps and stricter cutoffs, and the final NROY samples define the mean and variance of truncated-normal priors for ABC-SMC.

What would settle it

Take the calibrated posterior to an out-of-sample test: run the accepted parameters on new random seeds against a later wave (e.g., autumn 2020) or a neighboring county's data, and check whether the 90 percent predictive intervals contain the observed diagnoses and deaths. Alternatively, rerun history matching with a nonzero model-discrepancy term (say, a few percent of the observed values) and see whether the NROY space becomes empty or whether the final posterior shifts materially; if it does, the 99 percent volume reduction was driven by the zero-discrepancy assumption rather than by the data.

Watch

Extended reading notes

Core claim

The central claim is that a four-round history matching procedure, using heteroskedastic Gaussian process emulators trained on 50 maximin Latin hypercube designs per wave with 20–25 replicates, can shrink the non-implausible ('NROY') volume of Covasim's four-parameter calibration space from 100 percent to 0.82 percent, and that the remaining region contains parameterizations whose ABC posterior reproduces the observed epidemic trajectories on new random seeds. The paper presents this as a supplement to the original trajectory-oriented calibration: instead of collecting a library of parameter-seed pairs that match data, it estimates parameter settings expected to match data on average, with variance across seeds providing a stability measure. The authors also show the calibrated parameters reproduce the original test-trace-quarantine counterfactual, supporting the model's use for policy experiments.

Load-bearing premise

The whole pipeline assumes the simulator has zero systematic error relative to reality — the paper explicitly sets model discrepancy to zero because some previously published Covasim trajectories matched the data — so if the model's simplified transmission and behavior mechanisms cannot actually reproduce the real King County epidemic, history matching will wrongly rule out the true parameters and the posterior will be overconfident.

Editorial extensions

If this is right

  • Policy-oriented agent-based model calibration can be completed with an order of magnitude fewer simulator runs (5,300 vs. over 100,000), shrinking calibration time from weeks to days.
  • The method yields a posterior distribution over parameter settings rather than a library of matching runs, giving an estimate of which settings match on average and how much seed-to-seed variance remains.
  • Efficient calibration makes it feasible to revisit 'fixed' model parameters and to enrich behavioral mechanisms, such as the testing-odds-ratio proxy for health-seeking behavior.
  • The same pipeline transfers to other models built on the Covasim/Starsim framework, broadening its reach beyond COVID-19.
  • Both mean-focused calibration and trajectory-oriented optimization can be used together to build confidence in intervention counterfactuals like the test-trace-quarantine scenario.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the zero-discrepancy assumption is relaxed, the reported 99 percent volume reduction is likely an overestimate, and the posterior credible intervals could widen; a natural extension is to calibrate discrepancy from out-of-sample predictions or expert priors.
  • The weaker fit to active infections suggests the outputs chosen for emulation constrain what the method can rule out; adding a better-observed proxy for infection prevalence could change which regions survive.
  • Because the posterior is tuned to King County spring 2020, transferring parameters to other places or later variants would require re-running history matching; the NROY region could serve as a compact informative prior for such transfer learning.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper demonstrates a calibration workflow for the Covasim agent-based COVID-19 model in King County, combining history matching with heteroskedastic Gaussian process emulation and approximate Bayesian computation (ABC). Four rounds of history matching reduce the candidate parameter grid from 2,560,000 points to 21,114 NROY points, after which an ABC-SMC posterior is obtained using emulator draws. The authors then run 50 Covasim simulations at posterior parameter samples and report that the resulting diagnoses and deaths time series, as well as a policy counterfactual, match empirical data and qualitatively reproduce an earlier analysis by Kerr et al. The central efficiency claim is that this requires roughly 5,300 simulator runs, far fewer than the 100,000 runs used in the original calibration.

Significance. If the central claims hold, the paper offers a practically useful demonstration that expensive policy-oriented agent-based models can be calibrated with far fewer simulator runs than trajectory-oriented hyperparameter optimization, while still providing posterior distributions over parameters. The strengths of the paper include reproducible code, Docker and Zenodo artifacts, clearly documented run counts, and a concrete policy intervention case study. However, the validation of the posterior is currently qualitative and the ABC posterior is obtained from emulator draws rather than simulator runs, so the significance of the reported match to empirical data is conditional on additional quantitative checks.

major comments (5)
  1. [Results, 'High-Fidelity Out of Sample Matching is Possible with ABC Following a History Match'; Figure 4] The central claim that the ABC posterior 'matches empirical data across random seeds' is supported only by visual inspection of Figure 4, based on 50 new-seed simulator runs. No quantitative posterior predictive check is reported: no empirical coverage of the observed data within the plotted 50% or 90% intervals, no normalized RMSE, and no interval score. Because the claim is specifically about the posterior's ability to reproduce the observed time series, this is load-bearing. I request a quantitative PPC for diagnoses and deaths (for example, coverage fractions and interval scores), which would also resolve whether the zero-discrepancy assumption makes the intervals overconfident.
  2. [Methods, 'Emulation and History Matching'; implausibility measure equation] The implausibility measure sets both model discrepancy and observational error to zero, so all uncertainty in the denominator is emulator variance. The authors' justification, that 'many trajectories shown in Kerr et al. 2021 were able to satisfy empirical data', does not imply zero model discrepancy at every candidate parameterization, and the paper itself acknowledges that this choice removes more input locations. With zero discrepancy, history matching can rule out the true parameter region and the ABC posterior can be overconfident. I request a sensitivity analysis in which model discrepancy is varied over a plausible range, or an equivalent inflation factor, with a report of how NROY volumes and posterior intervals change.
  3. [Methods, 'Detailed calibration via hetGPy and Approximate Bayesian Computation'] The ABC-SMC sampler draws 'sample simulations from our hetGPy' rather than from the actual Covasim simulator. The posterior is therefore conditional on emulator predictions, and any emulator bias in the NROY region will translate directly into a biased posterior. Only 50 posterior samples are subsequently run through Covasim, and those are not used to correct the ABC acceptance. I request emulator validation at held-out NROY locations (for example, predictive RMSE or coverage against simulator outputs) and, ideally, a final ABC stage that uses simulator draws, or a quantitative demonstration that emulator error is negligible in the accepted region.
  4. [Results, 'High-Fidelity Out of Sample Matching is Possible with ABC Following a History Match'; Table 1] The text states that 'After three rounds of history matching, we removed over 99% of parameter space from consideration,' but Table 1 shows that after wave 3 the NROY sample is 54,848 of 2,560,000 grid points, that is, 2.14% remaining and 97.86% removed. The 'over 99%' claim holds only after the fourth wave (21,114 points, 0.82% remaining). The wording should be corrected, and the Figure 2 caption's reference to 'after three waves' should be checked for consistency.
  5. [Results, 'High-Fidelity Out of Sample Matching is Possible with ABC Following a History Match'] The title and the phrase 'out of sample' are misleading: the 50 posterior simulations use new random seeds, but they are evaluated against the same empirical data used to build the history-matching targets and the ABC posterior. This is a within-sample check with respect to the data, not an out-of-sample test. Rename this section or explicitly state that only stochastic seeds are new, and note that the fit in Figure 4 partly reflects fitting to the same data.
minor comments (4)
  1. [Results, first paragraph] The phrase 'the progression of simulator runs across history matching rounds in shown in Figure 1' contains a typo; it should read 'is shown in Figure 1'.
  2. [Figure 2 caption] The caption contains a duplicated phrase: 'estimating the remaining the remaining volume of NROY space' should read 'estimating the remaining volume of NROY space'.
  3. [Methods, 'Detailed calibration via hetGPy and Approximate Bayesian Computation'] The description of the ABC distance is ambiguous: the authors say they 'train two final hetGPy emulator models on the time-series of new diagnoses and deaths, respectively, and calibrate to empirical data using ABC,' but it is not clear whether diagnoses and deaths are calibrated separately or combined in a single Gaussian distance metric. Please specify the exact distance function, including any normalization or weighting, and whether one or two ABC analyses were performed.
  4. [Table 1] The column labeled 'NROY Samples' actually reports counts of grid points on the discrete 40^4 parameter grid, not Monte Carlo samples. Renaming this column to 'NROY grid points' would avoid confusion.

Circularity Check

1 steps flagged · score 2.0 of 10

Posterior 'out-of-sample' match is partly a restatement of ABC acceptance; the efficiency claim is independent.

  1. fitted input called prediction [Results, section 'High-Fidelity Out of Sample Matching is Possible with ABC Following a History Match' (Figure 4 paragraph)]
    "We tested the accuracy of our posterior distribution by simulating 50 samples. Our fits to empirical data are shown in Figure 4. We are able to estimate the time-series of diagnoses and death (two of the metrics shown in Kerr et. al 2021) without re-running simulations on the same seeds they were trained on. In this case, we used new random seeds during each history matching wave and for the final 50 posterior simulations."

    The ABC step is defined to retain parameters whose simulated diagnosis/death series are close to the observed King County empirical data (epsilon=5, Gaussian distance; 'we can keep parameters that generate outputs similar to observed empirical data'). The results section then presents 'fits to empirical data' as evidence for the 'High-Fidelity Out of Sample Matching' claim. The empirical series used for this evaluation are the same series used as the ABC acceptance target; only the random seed dimension is new. Thus the agreement in Figure 4 is partly a restatement of the ABC acceptance rule rather than a prediction against independent data, and no held-out empirical data or quantitative posterior predictive check (coverage, RMSE) is reported.

full rationale

The main methodological contribution—four rounds of history matching ruling out 99% of the parameter grid and completing calibration in 5,300 runs—is not circular: NROY volume is computed directly from emulator implausibility and compared with the previous 100,000-run calibration. The zero model-discrepancy and zero observational-error assumptions are explicit modeling choices, not hidden inputs, and although they affect the validity of the probability statements they do not make the derivation circular. The one overreach is the section title 'High-Fidelity Out of Sample Matching': the posterior was fitted to the same King County diagnosis/death time series used as the ABC target, so Figure 4's fit is an in-sample posterior predictive check, not an external prediction. The new-seed component is a genuine out-of-sample element, so this is a partial overstatement rather than a fully circular chain. No self-citation is load-bearing; the cited prior Covasim work is external peer-reviewed work.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; hetGPy is a software implementation, not an invented scientific entity. The method rests on standard statistical assumptions plus several domain assumptions about Covasim's adequacy, the zero-discrepancy ansatz, and the sufficiency of the chosen outputs.

free parameters (4)
  • Implausibility cutoff thresholds = 3.0, 2.7, 2.5
    Chosen by the authors starting from the standard 3-sigma rule and tightened each wave; directly controls the volume of NROY space and hence which parameters survive to the ABC stage.
  • ABC acceptance threshold (epsilon) = 5
    Chosen for the Gaussian distance metric without stated formal justification; determines the width of the accepted posterior and is central to the claim that posterior samples match data.
  • Replicates per design point = 25 in wave 1, 20 thereafter
    Chosen based on computing resources and the claim that 20 replicates model variance more robustly; affects the hetGP noise estimates that drive the implausibility measure.
  • Grid density and designs per wave = 40 grid points per dimension; 50 designs per wave
    The reported 2,560,000-point grid and the resulting NROY counts depend on these discretization choices, which are selected for resolution and computational feasibility.
assumptions (6)
  • domain assumption Covasim v2.1.2 with Synthpops contact networks is an adequate mechanistic representation of COVID-19 transmission in King County for the calibration period.
    The whole calibration treats simulator output as the data-generating process, as described in the 'Covasim' section.
  • domain assumption The model has zero discrepancy with reality and zero observational error for the chosen outputs.
    Stated in 'Emulation and History Matching': 'we assume no model discrepancy.' If false, implausibility thresholds over-rule the true parameter region.
  • domain assumption The heteroskedastic GP emulator provides unbiased predictions and correct uncertainty at unsampled points, including the 2,560,000-point grid.
    Used throughout to compute implausibility and to run ABC; no cross-validation or calibration checks are reported.
  • domain assumption Only the four parameters (beta, bc_wc1, bc_lf, tn) need to be uncertain; all other Covasim parameters are fixed at published values.
    Section 'Covasim'; if other parameters are mis-specified, the estimated posterior for the four parameters will absorb the error.
  • standard math Pukelsheim's three-sigma rule bounds at least 95% of a unimodal distribution within 3 standard deviations, justifying the I=3 cutoff.
    Cited as (Pukelsheim 1994); mathematical result used to set the implausibility cutoff.
  • domain assumption The chosen summary outputs (cumulative diagnoses and deaths at specific dates, active infections at two dates) are sufficient to constrain the parameters of interest.
    The calibration is defined by this output set; different output choices could lead to different NROY volumes and posteriors.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Policy-Oriented Agent-Based Modeling with History Matching: A Case Study." pith.science (2026). https://pith.science/paper/WHLZDRVW

@misc{pith2026250100616,
  author       = {Pith},
  title        = {Pith review of: Improving Policy-Oriented Agent-Based Modeling with History Matching: A Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WHLZDRVW}},
  note         = {Machine review of arXiv:2501.00616}
}
read the original abstract

Advances in computing power and data availability have led to growing sophistication in mechanistic mathematical models of social dynamics. Increasingly these models are used to inform real-world policy decision-making, often with significant time sensitivity. One such modeling approach is agent-based modeling, which offers particular strengths for capturing spatial and behavioral realism, and for in-silico experiments (varying input parameters and assumptions to explore their downstream impact on key outcomes). To be useful in the real world, these models must be able to qualitatively or quantitatively capture observed empirical phenomena, forming the starting point for subsequent experimentation. Computational constraints often form a significant hurdle to timely calibration and policy analysis in high resolution agent-based models. In this paper, we present a technical solution to address this bottleneck, substantially increasing efficiency and thus widening the range of utility for policy models. We illustrate our approach with a case study using a previously published and widely used epidemiological model.

Figures

Figures reproduced from arXiv: 2501.00616 by the authors.

Figure 1
Figure 1. Summary of Covasim model runs across rounds of history matching. Each color denotes a history matching round, where gray corresponds to the first, and green corresponds to the fourth. Examination of the non-implausible space via pairwise plots reveals the reduction in parameter space volume [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Optical depth plots of parameter pairs after three waves of history matching. Plots are generated by calculating the proportion of non-implausible points in parameter space that share each x,y coordinate, estimating the remaining the remaining volume of NROY space. As history matching rounds continue, we are able to train more accurate emulator models because we are searching over a much smaller region of parameter … view at source ↗
Figure 3
Figure 3. Progression of design locations (Covasim parameter simulations) across history matching waves. Note that “Wave 4” is the final round of history matching, which were the design locations for the ABC analysis. High-Fidelity Out of Sample Matching is Possible with ABC Following a History Match After three rounds of history matching, we removed over 99% of parameter space from consideration. The remaining parameter volu… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Fit to empirical data after history matching and emulation with hetGPy. hetGPy models were fit to the time-series of diagnoses and deaths and calibrated to empirical data via Sequential Monte Carlo Approximate Bayesian Computation. Panel (d) shows the 50 posterior samp…
Figure 5
Figure 5. Figure 5: Calibrated model parameters are capable of informing policy intervention scenarios. Simulations reproduce a counterfactual policy intervention discussed in Kerr et. al where King County expanded its diagnostic testing and contact tracing capacity to allow re-opening in…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 22 canonical work pages

  1. [1]

    trajectory-oriented optimization

    Improving Policy-Oriented Agent-Based Modeling with History Matching: A Case Study David O’Gara*1, Cliff C. Kerr2, Daniel J. Klein2, Mickaël Binois3, Roman Garnett1,4, Ross A. Hammond1,5,6,7 Affiliations 1Division of Computational and Data Sciences, Washington University in St. Louis, St. Louis, MO 2Institute for Disease Modeling, Bill & Melinda Gates Fou...

    work page 2006

  2. [3]

    beta”, the relative per-contact transmission reductions in work and community layers from March 23, 2020 onwards (“bc_wc1

    library, with four contact layers representing home, workplace, school, and community contacts. Like other agent-based COVID- 19 models, Covasim follows the Susceptible-Exposed-Infected-Recovered-Dead (SEIRD) model, with their discrete state describing their current health status, which also determines the probability of when and which state they will tra...

    work page 2020

  3. [5]

    Table 1: Summary of history matching rounds. Detailed calibration via hetGPy and Approximate Bayesian Computation Upon ruling out large region of the model parameter space, we expect that the remaining parameters may contain good fits to empirical data on average. On this smaller space, we train two final hetGPy emulator models on the time-series of new d...

    work page 2020

  4. [6]

    We also observe lower variance in model outputs in later rounds

    We see model runs become more accurate across subsequent rounds, especially for the time-series of diagnoses. We also observe lower variance in model outputs in later rounds. Fitting the time-series of deaths is challenging, due to the low number of deaths per day reflected in empirical data, but our fits to the cumulative number of deaths improves with i...

    work page 2021

  5. [7]

    In this case, we used new random seeds during each history matching wave and for the final 50 posterior simulations

    without re-running simulations on the same seeds they were trained on. In this case, we used new random seeds during each history matching wave and for the final 50 posterior simulations. Specifically, each random seed for a simulation corresponds to the order in which simulations were run (in this case, 0 to 5,350). Fitting to the number of estimated act...

    work page 2015

  6. [11]

    J UNE : Open-Source Individual-Based Epidemiology Simulation

    “J UNE : Open-Source Individual-Based Epidemiology Simulation.” Royal Society Open Science 8(7):210506. doi: 10.1098/rsos.210506. Baker, Evan, Pierre Barbillon, Arindam Fadikar, Robert B. Gramacy, Radu Herbei, David Higdon, Jiangeng Huang, Leah R. Johnson, Pulong Ma, Anirban Mondal, Bianica Pires, Jerome Sacks, and Vadim Sokolov

    doi:10.1098/rsos.210506

  7. [12]

    Analyzing Stochastic Computer Models: A Review with Opportunities

    “Analyzing Stochastic Computer Models: A Review with Opportunities.” doi: 10.48550/ARXIV .2002.01321. Beaumont, Mark A., Jean-Marie Cornuet, Jean-Michel Marin, and Christian P. Robert

    arXiv doi:10.48550/arxiv.2002.01321 2002

  8. [13]

    A Review and Agenda for Integrated Disease Models Including Social and Behavioural Factors

    “A Review and Agenda for Integrated Disease Models Including Social and Behavioural Factors.” Nature Human Behaviour 5(7):834–46. doi: 10.1038/s41562-021-01136-2. Bergstra, James, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl

    doi:10.1038/s41562-021-01136-2

Show all 35 references
  1. [16]

    Imperial College London

    Report 9: Impact of Non-Pharmaceutical Interventions (NPIs) to Reduce COVID19 Mortality and Healthcare Demand. Imperial College London. doi: 10.25561/77482. Ferguson, Neil M., Derek A. T. Cummings, Christophe Fraser, James C. Cajka, Philip C. Cooley, and Donald S. Burke

    doi

  2. [17]

    Strategies for Mitigating an Influenza Pandemic

    “Strategies for Mitigating an Influenza Pandemic.” Nature 442(7101):448–52. doi: 10.1038/nature04795. Garnett, Roman

    doi

  3. [18]

    Mitigation Strategies for Pandemic Influenza in the United States

    “Mitigation Strategies for Pandemic Influenza in the United States.” Proceedings of the National Academy of Sciences 103(15):5935. doi: 10.1073/pnas.0601266103. Gramacy, Robert B

    doi

  4. [21]

    OpenABM-Covid19—An Agent-Based Model for Non-Pharmaceutical Interventions against COVID-19 Including Contact Tracing

    “OpenABM-Covid19—An Agent-Based Model for Non-Pharmaceutical Interventions against COVID-19 Including Contact Tracing.” PLOS Computational Biology 17(7):e1009146. doi: 10.1371/journal.pcbi.1009146. Holthuijzen, Maike F., Robert B. Gramacy, Cayelan C. Carey, Dave M. Higdon, and...

    doi

  5. [22]

    Emulation and History Matching Using the Hmer Package

    “Emulation and History Matching Using the Hmer Package.” Journal of Statistical Software 109(10). doi: 10.18637/jss.v109.i10. Kerr, Cliff C., Dina Mistry, Robyn M. Stuart, Katherine Rosenfeld, Gregory R. Hart, Rafael C. Núñez, Jamie A. Cohen, Prashanth Selvaraj, Romesh G. Abey...

    doi

  6. [23]

    Controlling COVID-19 via Test-Trace-Quarantine

    “Controlling COVID-19 via Test-Trace-Quarantine.” Nature Communications 12(1):2993. doi: 10.1038/s41467-021-23276-9. Kerr, Cliff C., Robyn M. Stuart, Dina Mistry, Romesh G. Abeysuriya, Katherine Rosenfeld, Gregory R. Hart, Rafael C. Núñez, Jamie A. Cohen, Prashanth Selvaraj, B...

    doi

  7. [24]

    Covasim: An Agent-Based Model of COVID-19 Dynamics and Interventions

    “Covasim: An Agent-Based Model of COVID-19 Dynamics and Interventions.” PLOS Computational Biology 17(7):e1009149. doi: 10.1371/journal.pcbi.1009149. Kerr, Cliff, Robyn M. Stuart, Romesh G. Abeysuriya, Jamie A. Cohen, Paula Sanz-Leon, Alina Muellenmeister, and Daniel Klein. n....

    doi

  8. [26]

    TRACE‐Omicron: Policy Counterfactuals to Inform Mitigation of COVID‐19 Spread in the United States

    “TRACE‐Omicron: Policy Counterfactuals to Inform Mitigation of COVID‐19 Spread in the United States.” Advanced Theory and Simulations 2300147. doi: 10.1002/adts.202300147. Ozik, Jonathan, Justin M. Wozniak, Nicholson Collier, Charles M. Macal, and Mickaël Binois

    doi

  9. [27]

    A Population Data-Driven Workflow for COVID-19 Modeling and Learning

    “A Population Data-Driven Workflow for COVID-19 Modeling and Learning.” The International Journal of High Performance Computing Applications 35(5):483–99. doi: 10.1177/10943420211035164. Panovska-Griffiths, Jasmina, Thomas Bayley, Tony Ward, Akashaditya Das, Luca Imeneo, Cliff...

    doi

  10. [28]

    preprint

    Machine Learning Assisted Calibration of Stochastic Agent-Based Models for Pandemic Outbreak Analysis. preprint. In Review. doi: 10.21203/rs.3.rs-2773605/v1. Pukelsheim, Friedrich

    doi

  11. [29]

    The Three Sigma Rule

    “The Three Sigma Rule.” The American Statistician 48(2):88–91. doi: 10.1080/00031305.1994.10476030. Reiker, Theresa, Monica Golumbeanu, Andrew Shattock, Lydia Burgert, Thomas A. Smith, Sarah Filippi, Ewan Cameron, and Melissa A. Penny

    1994 arXiv

  12. [30]

    Emulator-Based Bayesian Optimization for Efficient Multi-Objective Calibration of an Individual-Based Model of Malaria

    “Emulator-Based Bayesian Optimization for Efficient Multi-Objective Calibration of an Individual-Based Model of Malaria.” Nature Communications 12(1):7212. doi: 10.1038/s41467-021-27486-z. Santner, Thomas J., Brian J. Williams, and William I. Notz

    doi

  13. [33]

    Bayesian Emulation and History Matching of JUNE

    “Bayesian Emulation and History Matching of JUNE.” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 380(2233):20220039. doi: 10.1098/rsta.2022.0039. Vernon, Ian, Michael Goldstein, and Richard G. Bower

    2022

  14. [35]

    Bayesian Uncertainty Analysis for Complex Systems Biology Models: Emulation, Global Parameter Searches and Evaluation of Gene Functions

    “Bayesian Uncertainty Analysis for Complex Systems Biology Models: Emulation, Global Parameter Searches and Evaluation of Gene Functions.” BMC Systems Biology 12(1):1. doi: 10.1186/s12918-017-0484-3. Improving Policy-Oriented Agent-Based Modeling with History Matching: A Case ...

    doi

  15. [1994]

    2015; Iskauskas, Vernon, et al

    showing that at least 95% of any continuous unimodal distribution is contained within three standard deviations and has been used in the history matching literature (Andrianakis et al. 2015; Iskauskas, Vernon, et al. 2024; Vernon et al. 2022, 2010). Emulation via hetGPy Our ch...

    2015

  16. [2005]

    Containing Pandemic Influenza at the Source

    “Containing Pandemic Influenza at the Source.” Science 309(5737):1083–87. doi: 10.1126/science.1115717. Mistry, Dina, and C. C. Kerr

    doi

  17. [2006]

    Individual-Based Computational Modeling of Smallpox Epidemic Control Strategies

    “Individual-Based Computational Modeling of Smallpox Epidemic Control Strategies.” Academic Emergency Medicine 13(11):1142–49. doi: 10.1197/j.aem.2006.07.017. Fadikar, Arindam, Mickael Binois, Nicholson Collier, Abby Stevens, Kok Ben Toh, and Jonathan Ozik

    2006 doi

  18. [2007]

    Sequential Monte Carlo without Likelihoods

    “Sequential Monte Carlo without Likelihoods.” Proceedings of the National Academy of Sciences 104(6):1760–65. doi: 10.1073/pnas.0607208104. Sun, Furong, and Robert B. Gramacy

    doi

  19. [2008]

    Modeling Targeted Layered Containment of an Influenza Pandemic in the United States

    “Modeling Targeted Layered Containment of an Influenza Pandemic in the United States.” Proceedings of the National Academy of Sciences 105(12):4639–44. doi: 10.1073/pnas.0706849105. Hammond, Ross A

    doi

  20. [2010]

    Galaxy Formation: A Bayesian Uncertainty Analysis

    “Galaxy Formation: A Bayesian Uncertainty Analysis.” Bayesian Analysis 5(4):619–69. doi: 10.1214/10-BA524. Vernon, Ian, Junli Liu, Michael Goldstein, James Rowe, Jen Topping, and Keith Lindsey

    doi

  21. [2015]

    Bayesian History Matching of Complex Infectious Disease Models Using Emulation: A Tutorial and a Case Study on HIV in Uganda

    “Bayesian History Matching of Complex Infectious Disease Models Using Emulation: A Tutorial and a Case Study on HIV in Uganda.” PLOS Computational Biology 11(1):e1003968. doi: 10.1371/journal.pcbi.1003968. Aylett-Bullock, Joseph, Carolina Cuesta-Lazaro, Arnau Quera-Bofarull, M...

    doi

  22. [2018]

    Practical Heteroscedastic Gaussian Process Modeling for Large Simulation Experiments

    “Practical Heteroscedastic Gaussian Process Modeling for Large Simulation Experiments.” Journal of Computational and Graphical Statistics 27(4):808–21. doi: 10.1080/10618600.2018.1458625. Burke, Donald S., Joshua M. Epstein, Derek A. T. Cummings, Jon I. Parker, Kenneth C. Clin...

    2018

  23. [2020]

    status quo

    Our analysis in Figure 5 also comports with these findings when we use our calibrated model parameters and project them forward in time. We do observe that the simulations from the original work (labeled TTQ) appear to follow the data more closely, while our method (ABC) conta...

    2020

  24. [2021]

    al 2021 and the methods paper (Kerr, Stuart, et al

    Covasim’s functionality is described in detail in both Kerr et. al 2021 and the methods paper (Kerr, Stuart, et al. 2021), but we briefly describe the core components here. Covasim is a large-scale agent-based stochastic transmission model. The present analysis with Covasim ve...

    2021

  25. [2022]

    Impact of Vaccination and Non-Pharmaceutical Interventions on SARS-CoV-2 Dynamics in Switzerland

    “Impact of Vaccination and Non-Pharmaceutical Interventions on SARS-CoV-2 Dynamics in Switzerland.” Epidemics 38:100535. doi: 10.1016/j.epidem.2021.100535. Sisson, S. A., Y . Fan, and Mark M. Tanaka

    2021

  26. [2023]

    PyMC: A Modern, and Comprehensive Probabilistic Programming Framework in Python

    “PyMC: A Modern, and Comprehensive Probabilistic Programming Framework in Python.” PeerJ Computer Science 9:e1516. doi: 10.7717/peerj-cs.1516. Akiba, Takuya, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama

    doi

  27. [2024]

    Making Evidence Go Further: Advancing Synergy between Agent-Based Modeling and Randomized Control Trials

    “Making Evidence Go Further: Advancing Synergy between Agent-Based Modeling and Randomized Control Trials.” Proceedings of the National Academy of Sciences 121(21):e2314993121. doi: 10.1073/pnas.2314993121. Hammond, Ross, Joseph T. Ornstein, Rob Purcell, Matthew D. Haslam, and...

    doi

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.