REVIEW 2 minor 19 references
De-meaning Simulation Studies
T0 review · 0 major / 2 minor · reviewed 2026-06-26 · grok-4.3
Pith's one-line read Simulation studies evaluating distributional approximations should report medians and coverage rates instead of means and standard deviations.
desk verdict The paper makes a straightforward case for switching to quantiles when simulations check distributional approximations rather than moments themselves. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The distinction between convergence in distribution and convergence of moments, together with the practical fragility of means in the presence of outliers; this distinction motivates the shift to median, median absolute deviation, and empirical coverage as the default reporting tools.
What would settle it
A concrete simulation in which the target distribution converges in law but the sample mean and variance across replications diverge or become unstable, while the median, MAD, and coverage remain stable and correctly indicate good approximation quality.
Extended reading notes
Core claim
Quantile-based summaries are more appropriate than moment-based ones for assessing the accuracy of distributional approximations in simulation studies. Theoretically, convergence in distribution does not imply convergence of moments or even their existence, so sample moments are not guaranteed to reflect the quality of the approximation. Practically, means and variances are sensitive to outliers even when the distribution is approximately normal. The paper therefore recommends the median and median absolute deviation as general summaries, together with empirical confidence-interval coverage, and reserves moments for cases where they are the direct object of substantive interest.
Load-bearing premise
The main goal of the simulation study is to judge how well a distributional approximation works, rather than to study moments that have direct substantive meaning.
Editorial extensions
If this is right
- Papers claiming asymptotic normality would report median absolute deviation rather than standard deviation to describe variability across simulations.
- Empirical coverage of nominal confidence intervals would become a standard reported quantity in simulation tables.
- Moments would appear in simulation results only when the study is explicitly investigating expected values or variances as quantities of interest.
- Outlier-resistant summaries would reduce the influence of rare simulation failures on reported performance.
Reading between the lines
- Re-analysis of existing simulation studies that used means could change conclusions about how well certain approximations perform.
- Software packages for Monte Carlo simulation could adopt quantile summaries as the default output format.
- The same logic extends to simulation studies in other fields that validate limiting distributions, such as bootstrap or MCMC diagnostics.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript argues that simulation studies evaluating asymptotic approximations should prefer quantile-based summaries (median, median absolute deviation, and empirical coverage) over means and standard deviations. The theoretical rationale is that convergence in distribution does not imply convergence or even existence of moments, rendering sample moments unsuitable for assessing distributional accuracy. Practically, means and variances are sensitive to occasional outliers even when the underlying distribution is approximately normal. Moments are to be reserved for settings where they are the direct object of substantive interest.
Significance. The note identifies a widespread but theoretically unsupported reporting convention in simulation studies. Adoption would align reporting practices more closely with the actual inferential target (distributional approximation) and reduce sensitivity to tail behavior. The argument relies entirely on standard probability theory with no new parameters, derivations, or self-referential constructs.
minor comments (2)
- The manuscript is a short note; a single illustrative numerical example (even a small Monte Carlo illustration of mean vs. median behavior under a heavy-tailed approximation) would make the practical claim more concrete without altering the central argument.
- The title 'De-meaning Simulation Studies' is concise but may not immediately signal the content to readers scanning tables of contents; a subtitle or clearer phrasing could improve discoverability.
Simulated Author's Rebuttal
We thank the referee for their supportive review and recommendation to accept the manuscript. The summary accurately captures our central arguments on the theoretical and practical limitations of moment-based summaries in simulation studies of asymptotic approximations.
Circularity Check
No significant circularity
full rationale
The manuscript contains no equations, fitted parameters, or derivations. Its central recommendation—that quantile-based summaries are preferable for assessing distributional approximations in simulation studies—rests on the standard external fact that convergence in distribution does not imply convergence of moments, plus the practical observation that means and variances are outlier-sensitive. The paper explicitly scopes its advice to cases where the simulation goal is distributional accuracy rather than direct moment estimation, and invokes no self-citations, ansatzes, or uniqueness claims. The argument is therefore self-contained against external benchmarks and exhibits no reduction of outputs to inputs by construction.
Assumptions & free parameters
assumptions (1)
- standard math Convergence in distribution does not imply convergence or existence of moments
Cite this review
Pith. "Pith review of De-meaning Simulation Studies." pith.science (2026). https://pith.science/paper/LHCKWO7R
@misc{pith2026260621038,
author = {Pith},
title = {Pith review of: De-meaning Simulation Studies},
year = {2026},
howpublished = {\url{https://pith.science/paper/LHCKWO7R}},
note = {Machine review of arXiv:2606.21038}
}
read the original abstract
In simulation studies evaluating asymptotic approximations it is common practice to report averages and standard deviations over repeated simulations. We argue that quantile-based summaries are more appropriate from both a theoretical and practical point of view. Theoretically, convergence of moments -- or even existence of moments -- is not guaranteed by convergence in distribution, so sample moments are not ideal for assessing the accuracy of a distributional approximation. In practice, means and variances are not good summaries of approximately-Normal distributions that may have occasional outliers. We suggest the median and median absolute deviation, and empirical confidence interval coverage, as better general summaries, and argue that moments should be reserved for simulation settings where they are of substantive interest.
Reference graph
Works this paper leans on
-
[1]
Peter J Huber , title =
-
[2]
Proschan and Pamela A
Michael A. Proschan and Pamela A. Shaw , title =
-
[3]
Machine Learning , volume=
Random forests , author=. Machine Learning , volume=
-
[4]
Robust statistics: the approach based on influence functions (Wiley series in probability and statistics)
Hampel, Frank R and Ronchetti, Elvezio M and Rousseeuw, Peter J and Stahel, Werner A , address =. Robust statistics: the approach based on influence functions (Wiley series in probability and statistics). , volume =
-
[5]
Environmetrics , volume =
Lumley, Thomas and Levy, Drew , title =. Environmetrics , volume =
-
[6]
Journal of the American statistical Association , volume=
Calibration estimators in survey sampling , author=. Journal of the American statistical Association , volume=
-
[7]
The International Journal of Biostatistics , volume=
A targeted maximum likelihood estimator for two-stage designs , author=. The International Journal of Biostatistics , volume=
-
[8]
2003 , publisher=
Model Assisted Survey Sampling , author=. 2003 , publisher=
2003
Show all 19 references
-
[9]
Statistical Methods in Medical Research , volume=
Review of inverse probability weighting for dealing with missing data , author=. Statistical Methods in Medical Research , volume=
-
[10]
mice: Multivariate imputation by chained equations in
Van Buuren, Stef and Groothuis-Oudshoorn, Karin , journal=. mice: Multivariate imputation by chained equations in
-
[11]
Statistics in Medicine , year=
Assessing treatment effects in observational data with missing confounders: A comparative study of practical doubly-robust and traditional missing data methods , author=. Statistics in Medicine , year=. doi:10.1002/sim.70366 , volume=
-
[12]
Simmons and Leif D
Joseph P. Simmons and Leif D. Nelson and Uri Simonsohn , title =. Psychological Science , volume =. 2011 , doi =
2011
-
[13]
Journal of the American Statistical Association , volume =
Nir Keret and Malka Gorfine , title =. Journal of the American Statistical Association , volume =. 2025 , publisher =
2025
-
[14]
Journal of the American Statistical Association , volume =
Doudou Zhou and Molei Liu and Mengyan Li and Tianxi Cai , title =. Journal of the American Statistical Association , volume =. 2025 , publisher =
2025
-
[15]
and Yates, Philip A
Grego, John M. and Yates, Philip A. , date =. Robust Local Likelihood Estimation for Non-stationary Flood Frequency Analysis , url =. Journal of Agricultural, Biological and Environmental Statistics , number =. doi:10.1007/s13253-024-00614-0 , id =
-
[16]
Haider and Melvin
Steven J. Haider and Melvin. Correcting for Misclassified Binary Regressors Using Instrumental Variables , journal =. 2025 , publisher =
2025
-
[17]
Matteson , title =
Maryclare Griffin and Gennady Samorodnitsky and David S. Matteson , title =. Technometrics , volume =. 2025 , publisher =
2025
-
[18]
The Annals of Statistics , number =
Kosuke Morikawa and Yoshikazu Terada and Jae Kwang Kim , title =. The Annals of Statistics , number =. 2025 , doi =
2025
-
[19]
Biometrika , volume =
Bannick, Marlena S and Shao, Jun and Liu, Jingyi and Du, Yu and Yi, Yanyao and Ye, Ting , title =. Biometrika , volume =. 2025 , month =. doi:10.1093/biomet/asaf029 , url =
2025 doi
Reviewed June 26, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.