{"id":"1482ecbd-0e09-400e-83ef-08c82c17b9d9","arxiv_id":"2501.03138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MCBench is a modular Julia benchmark suite that scores Monte Carlo samplers by comparing their samples against IID reference samples using metrics like sliced Wasserstein distance and maximum mean discrepancy.","lead":"MCBench is a new Julia package that checks how well a Monte Carlo sampling algorithm reproduces a known target distribution by comparing its output with high-quality independent samples. It bundles several statistical distance measures into one workflow, so researchers can get a quantitative quality score instead of eyeballing histograms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Default ESS estimator in §2 uses the weight-based Kish formula; for unweighted MCMC output it returns the raw sample count, so the batch-based comparison bands in §6.1 are miscalibrated for autocorrelated samplers, undermining the central quantitative comparison claim.","rationale":"The paper is a useful, modular benchmark suite with a clear workflow and sensible metric choices (SWD, MMD, basic statistics). The core algorithm is transparent and the code is available. However, the central claim that users obtain 'clear, quantitative measures of sampling quality' depends critically on the calibration of the reference distributions. That calibration in turn depends on partitioning autocorrelated sampler output into batches with a correct effective sample size. The default ESS estimator in Section 2 is the Kish importance-weight formula; applied to unweighted MCMC chains it returns the raw sample count and ignores autocorrelation. Section 6.1 explicitly observes the resulting variance inflation. Because the batched metric distributions are the basis for the colored bands and the marker placement, a miscalibrated ESS can shift a sampler's apparent quality relative to the bands. This is not a minor implementation detail: the default workflow in Section 5 uses this estimator, and the primary example (Metropolis-Hastings) exhibits the problem. The paper does not provide an alternative calibration or a sensitivity analysis. I therefore agree with the reader's identification of the ESS estimator as the weakest assumption. The concern is testable: rerunning the walkthrough with an autocorrelation-aware ESS should change the bandwidth of the metric distributions if the default is wrong. This does not require rejecting the paper; it requires a methodological fix or a documented warning, so a conditional verdict is appropriate. Minor issues (the 3D/10D inconsistency in §6.2, the under-supported novelty claim, and the unvalidated accept-reject sampler for the eight schools case) are secondary.","tokens_in":15751,"tokens_out":8581,"duration_ms":79352,"concrete_test":"Repeat §6.1 with a sampler chain of known autocorrelation, e.g., a Gaussian AR(1) process tuned to match the BATMH chain's lag-1 autocorrelation on the 3D standard normal target. Build the test statistic twice: once with the default ESS (which returns the raw count for unweighted samples) and once with an autocorrelation-aware ESS estimator (e.g., n/(1+2Σρ̂_k) with a proper lag window). Compare the standard deviation of the marginal_mean metric across batches. If the default workflow yields a bandwidth that differs by more than ~20% from the autocorrelation-aware workflow, the default ESS assumption is falsified and the normalized bands in Fig. 3 are miscalibrated for MCMC output.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that MCBench yields reliable, normalized metric distributions for assessing sampler quality. That claim depends on partitioning user samples into m batches of effective size n (§2). The default ESS estimator defined in §2 is the ratio of the squared sum of weights divided by the sum of their squares, i.e., the Kish effective sample size for importance sampling. For the unweighted output of MCMC samplers, all weights are 1, so this estimator returns the raw sample count, regardless of autocorrelation. Section 6.1 explicitly notes an overestimation of ESS for the Metropolis-Hastings chain. As a consequence, each batch has true effective size smaller than n, and the per-batch metric distribution for the user sampler is wider than the corresponding IID null distribution even when the sampler's stationary distribution is exact. The normalized bands (1σ/2σ/3σ) are computed from IID batches of size n, so the comparison marker and error bars for a good but autocorrelated sampler can fall outside the colored regions, producing a false negative for sampling quality. This affects every MCMC evaluation in the benchmark, not just the walkthrough, and directly undermines the goal of 'clear, quantitative measures of sampling quality' (Abstract). Because the ESS estimate is a required input to the batch construction, a miscalibrated default makes the central comparison method unreliable for the primary use case (MCMC). The paper provides no validation of the default ESS estimator against known per-batch calibration, and the code path in §5 uses this default. This is a load-bearing correctness risk, not a consensus disagreement.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MCBench, a Julia package for benchmarking Monte Carlo sampling algorithms. Users select one of several IID-sampleable target distributions, draw samples with an external sampler, and the package compares those samples against IID samples from the same target using a set of metrics: basic statistics (marginal mean, marginal variance, chi-square), the sliced Wasserstein distance (SWD), and the maximum mean discrepancy (MMD), optionally with random Fourier features. The comparison is visualized by normalizing the IID-batch metric distributions to mean zero and coloring the 1-sigma, 2-sigma, and 3-sigma regions, then overlaying the user sampler's metric mean and spread. Two walkthrough examples with the BAT.jl Metropolis-Hastings sampler are given: a 3D standard normal and a mixture of correlated normals. The appendix contains algorithms and theoretical background for SWD and MMD.","tokens_in":16026,"tokens_out":7897,"duration_ms":66530,"significance":"If correctly calibrated, MCBench would fill a practical need for a domain-neutral, modular, and easy-to-use benchmark suite for Monte Carlo samplers. The paper provides an open-source Julia package, clear workflow diagrams, example implementations, and reproducible walkthroughs, which are valuable strengths. The theoretical appendices for SWD and MMD are standard and appear correct in their main claims. However, the default effective-sample-size estimator is miscalibrated for autocorrelated samplers, and the paper explicitly admits this in Section 6.1. Since the central purpose of the benchmark is to provide quantitative, reliable comparisons of sampling quality, this issue is load-bearing: the comparison bands are built from IID batches of nominal size n, while batches from MCMC samplers have a smaller true effective size, so the method can produce false negatives even for samplers that are exact at stationarity. The paper should be revised to correct this calibration problem before the quantitative comparison claims are supported.","major_comments":[{"comment":"The default effective-sample-size estimator defined in Section 2 as the ratio of the squared sum of the sample weights divided by the sum of their squares is the Kish formula; for unweighted MCMC output it returns the raw sample count, independent of autocorrelation. Section 6.1 states that the observed overestimation of the metric variance is due to an overestimation of the effective sample size for the Metropolis-Hastings chain. Consequently, the user-sample batches are not of effective size n, while the IID reference bands are computed from batches of n independent samples. Even a sampler that is exact in stationarity will therefore have per-batch metric distributions that are wider than the colored bands, producing systematic false negatives for every autocorrelated sampler evaluated with the default settings. This directly undermines the central claim of enabling reliable, quantitative measures of sampling quality. The fix is to use an autocorrelation-aware ESS estimator (e.g., batch means or a spectral-variance estimator) as the default, to require the user to supply a valid ESS, or to construct both the IID and user-sample batches using the same effective-size definition; a validation study demonstrating the calibration on samplers with known ESS would also be needed.","section":"§2, §6.1"},{"comment":"The workflow for the advanced two-sample metrics (SWD and MMD) is described in Section 2 as comparing IID batches directly with user-generated samples, without specifying how the effective sample size is handled. SWD and MMD estimates depend on the number of samples and on autocorrelation: a correlated sampler with the same raw count as an IID batch carries less independent information, so its metric values will have higher variance. If the user samples are not thinned or weighted to the same effective size as the IID reference batches, the comparison is miscalibrated in the same way as for the basic metrics. The manuscript should specify the effective-sample-size treatment for two-sample metrics, or explicitly state that only raw sample counts are matched and calibrate the bands accordingly.","section":"§2 (two-sample metrics)"}],"minor_comments":[{"comment":"The heading 'Sliced W assertein Distance' contains a typo; it should be 'Sliced Wasserstein Distance'.","section":"§4.2.1 heading"},{"comment":"The code defines a 10-dimensional mixture (MvNormal(r*ones(10), ones(10,10)*0.9 + I(10)*0.1) and names it 'normal_3d_multimodal_10std'), but the section title and text say 'in 3D'. Please clarify the dimension used in the walkthrough.","section":"§6.2"},{"comment":"The parameters n=100 and n_samples=10^5 are not clearly defined; the text interprets them as 100 batches of 10^5 samples each, but the parameter names suggest n might be the batch size and n_samples the total number of samples. Please document the parameter semantics in Section 5.","section":"§6.1 code"},{"comment":"In the RFF Monte Carlo approximation, the formula 'k(x,y) ≈ 1/D Σ 2 cos(ω^T + b_i)' is missing the feature arguments; it should read cos(ω_i^T x + b_i) cos(ω_i^T y + b_i).","section":"Appendix B.2"},{"comment":"The spectral density of the Gaussian kernel is stated inconsistently: Algorithm 3 uses ω ∼ N(0, σ^{-2}I), while the text first writes p(ω)=N(0,2σ^{-2}I) and then gives a density with exponent −∥ω∥^2 σ^2/2, which matches neither. Please unify the Fourier-domain normalization.","section":"Appendix B.2"},{"comment":"The Eight Schools entry shows the likelihood but not the prior distributions from the text; including the full model in the table or caption would make the table self-contained.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the miscalibrated default effective-sample-size estimator, which the authors themselves acknowledge in Section 6.1. This is a fixable problem, but it is central enough that the quantitative comparison claims are not currently supported. I see no indication of novelty or attribution problems; the manuscript appropriately credits existing packages and related work. The paper's scope fits the journal as a software/benchmark contribution, but the revision needs to address the ESS calibration and provide a validation study before acceptance is warranted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"MCBench is a genuinely useful software paper with one load-bearing methodological flaw. The new thing is the integrated package: a modular Julia suite that compares arbitrary sampler output to internally generated IID reference samples across several metrics (sliced Wasserstein, MMD, basic statistics) and reports normalized comparison bands. That fills a practical gap, and the design is clean: test case, sampler, and metric abstractions are separated, and the walkthrough makes the intended workflow concrete. The appendix math is standard and correctly cited.\n\nThe soft spot is the default effective sample size estimator. Section 2 defines ESS as the Kish ratio of squared weight sum over sum of squared weights. For unweighted MCMC output that returns the raw sample count, so autocorrelation is ignored. Section 6.1 admits this: the MH walkthrough shows metric variance inflated relative to the IID bands. That is not a minor blemish. The colored 1σ/2σ/3σ bands are built from IID batches of size n, so an autocorrelated sampler whose stationary distribution is correct will produce per-batch metric distributions that are wider than the bands. The marker can land outside the green region even though the sampler is fine, which defeats the central claim of providing clear quantitative sampling-quality measures for MCMC. The fix is straightforward—use a proper MCMC ESS (e.g., bulk-ESS or a lag-based estimate) as the default, or require it as input—but as written the default can mislead.\n\nThere are smaller issues. Section 3 promises a one-dimensional Gaussian walkthrough; Section 6.1 actually uses a 3D standard normal. Section 6.2's code creates 10-dimensional mixture components while the text says 3D. The introduction's claim that no general domain-neutral tool exists is asserted without a serious survey of existing benchmark ecosystems. These are fixable and not as important as the ESS problem.\n\nWho should read it: anyone developing or evaluating MCMC samplers who wants a quick relative comparison across targets. It deserves peer review—the package is useful and the idea is sound—but the referee should require the ESS methodology to be fixed and validated, for instance with calibration experiments on chains with known autocorrelation. I would referee it.","headline":"Useful modular benchmark suite, but the default ESS estimator miscalibrates the comparison bands for MCMC output, which is the main use case.","tokens_in":16591,"tokens_out":4546,"would_cite":false,"duration_ms":37182,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["65C05","62F15","62G10"],"pacs":[],"model":"deepseek-v4-flash","headline":"MCBench is a Julia benchmark suite that claims to make Monte Carlo sampling quality quantitatively comparable by pitting user samples against IID reference samples from known targets using sliced Wasserstein distance and maximum mean…","keywords":["Monte Carlo sampling","benchmarking","sliced Wasserstein distance","maximum mean discrepancy","effective sample size","Markov chain Monte Carlo","Julia package","sample quality"],"falsifier":"Re-run the 3D Gaussian walkthrough twice: once with the default weight-based effective sample size estimator and once with an autocorrelation-aware estimator, for example one based on integrated autocorrelation time. If the Metropolis-Hastings metric markers shift substantially relative to the 1-, 2-, and 3-sigma IID bands, the suite's normalized comparison depends on the choice of ESS estimator rather than on sampling quality alone. A second check: run a sampler that provably misses one mode of the correlated mixture target and verify that the SWD and MMD markers fall outside the 3-sigma bands; if they do not, the metrics are not sensitive enough to detect the failure the paper claims they expose.","tokens_in":15547,"feed_emoji":"🎲","tokens_out":8952,"duration_ms":75369,"temperature":0.7,"pith_summary":"MCBench is a Julia benchmark suite that attempts to turn 'how good is my Monte Carlo sampler?' from a visual judgement into a quantitative comparison. Its method is to generate true independent-and-identically-distributed (IID) samples from a set of known test targets, then compare user-supplied samples from the same targets against those references using both basic statistics and two advanced metrics: the sliced Wasserstein distance and the maximum mean discrepancy. The comparison is repeated on many batches of equal effective sample size, so each metric yields a distribution of values rather than a single number; user-sampler values are then plotted against the IID-derived mean and 1, 2, and 3 standard-deviation bands. If the protocol works, sampler developers and users can benchmark any external algorithm and see at a glance whether their samples fall inside the scatter expected of true IID samples.","feed_headline":"MCBench puts Monte Carlo samplers to the IID test","feed_subtitle":"Sliced Wasserstein distance and MMD build normalized bands that show whether a sampler reproduces the target.","key_machinery":"The load-bearing mechanism is the batch-comparison protocol. Each sample set is divided into $m$ batches of effective sample size $n$; the default effective sample size estimator is the ratio of the squared sum of the weights to the sum of squared weights, which is intended to make autocorrelated sampler output comparable to $n$ IID samples. For each batch, the suite computes basic metrics (marginal mean, variance, chi-square) and, for two-sample comparisons, the sliced Wasserstein distance and the maximum mean discrepancy. The sliced Wasserstein distance projects both distributions onto random directions and averages the one-dimensional Wasserstein distances; the maximum mean discrepancy compares mean embeddings in a Gaussian reproducing kernel Hilbert space, computed exactly or via random Fourier features. The distributions of metric values over the IID batches define the expected scatter and the normalization bands used to judge the user-sampler batches.","core_discovery":"The central claim is that MCBench provides a general, domain-neutral workflow for evaluating Monte Carlo sampling algorithms. The user picks a target from a list of IID-sampleable distributions (including correlated and multimodal Gaussians, Cauchy, and the eight-schools hierarchical model), generates samples with the sampler of their choice, and imports them into the suite. MCBench splits both the user samples and its own IID reference samples into $m$ batches of effective sample size $n$, evaluates each metric on every batch, and builds the reference distribution of metric values from the IID batches. The user-sampler batches are then normalized relative to that IID distribution, so the final plots show whether the sampler's metric mean sits inside the green, yellow, or red bands. The paper demonstrates the workflow with the Metropolis-Hastings sampler from BAT.jl: on an uncorrelated 3D Gaussian the metrics match the IID expectation, while on a correlated mixture model the metric distributions clearly deviate, showing the suite can expose a sampler's failure to capture multimodal structure.","pith_inferences":["The IID-vs-IID metric distribution functions as an empirical null distribution, so one could assign an empirical p-value to a user sampler's metric value; the paper displays bands but does not compute p-values.","Replacing the default effective sample size estimator with an autocorrelation-aware one would likely widen the user-sampler bands for MCMC output, changing how close to the IID bands a sampler appears; this is a direct, testable consequence of the limitation admitted in Section 6.1.","The same machinery could rank approximate inference methods beyond MCMC, such as variational approximations or normalizing flows, as long as they emit samples; the paper targets MC sampling but the comparison protocol does not depend on how the user sample was produced.","MMD with a Gaussian kernel is a kernel two-sample test, so MCBench could be extended from distance reporting to calibrated power analysis, answering 'which metric detects a given sampler failure at which sample size?' rather than only 'how far is this sampler from IID?'"],"forward_implications":["Users can benchmark external samplers by writing samples to a CSV or HDF5 file; no integration with MCBench's sampling code is required.","The normalized bands give a thresholded visual test: a sampler whose metric mean falls outside the 3-sigma region fails to reproduce the target according to that metric.","Because every target is IID-sampleable, any detected discrepancy is attributable to the sampler rather than to reference-sample noise.","The modular TestCase-Sampler-Metric design means new target functions and new distance metrics can be added without changing the comparison protocol.","The eight-schools target provides a realistic hierarchical model with three prior levels, letting users test samplers on a nontrivial posterior where an IID accept-reject reference is still available."],"supporting_citations":[{"why":"Supplies the Metropolis algorithm, the random-walk MCMC method whose Metropolis-Hastings variant is used in the walkthrough examples.","marker":"[3]"},{"why":"Provides the BAT.jl Metropolis-Hastings sampler and DensitySampleVector interface used to generate the user samples under test.","marker":"[5]"},{"why":"Supplies the eight-schools hierarchical model used as a realistic test case with an accept-reject IID reference.","marker":"[7]"},{"why":"Gives the definition of the p-Wasserstein distance on which the sliced Wasserstein distance is based.","marker":"[12]"},{"why":"Introduces the random-projection sliced Wasserstein distance that the suite uses for high-dimensional comparisons.","marker":"[14]"},{"why":"Defines maximum mean discrepancy in a reproducing kernel Hilbert space, including the kernel expansion the suite implements.","marker":"[15]"},{"why":"Justifies the random Fourier features used to approximate the Gaussian-kernel MMD at large sample sizes.","marker":"[18]"},{"why":"Provides the Distributions.jl objects used to define test targets and generate IID reference samples.","marker":"[19, 20]"}],"fun_headline_variants":["MCBench: benchmark suite for Monte Carlo sampling algorithms","MCBench: quantifying Monte Carlo sample quality","MCBench: the IID test for Monte Carlo samplers","MCBench: measure your Monte Carlo against IID","MCBench: sliced Wasserstein and MMD for MC samples"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the default effective sample size estimator correctly splits a sampler's output into batches that behave like independent samples; the paper itself notes in Section 6.1 that this estimator overestimates the effective sample size for the Metropolis-Hastings chain, so the comparison bands could be narrower than they should be and the verdict could flatter a correlated sampler.","fun_headline_variants_meta":{"raw":{"variants":["MCBench: benchmark suite for Monte Carlo sampling algorithms","MCBench: quantifying Monte Carlo sample quality","MCBench: the IID test for Monte Carlo samplers","MCBench: measure your Monte Carlo against IID","MCBench: sliced Wasserstein and MMD for MC samples"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000268,"raw_usage":{"total_tokens":1656,"prompt_tokens":1020,"completion_tokens":636,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":636,"completion_tokens_details":{"reasoning_tokens":557}},"tokens_in":636,"tokens_out":636,"duration_ms":6228,"temperature":1.0,"reasoning_tokens":557,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:52:39.252433+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 3D Gaussian walkthrough twice: once with the default weight-based effective sample size estimator and once with an autocorrelation-aware estimator, for example one based on integrated autocorrelation time. If the Metropolis-Hastings metric markers shift substantially relative to the 1-, 2-, and 3-sigma IID bands, the suite's normalized comparison depends on the choice of ESS estimator rather than on sampling quality alone. A second check: run a sampler that provably misses one mode of the correlated mixture target and verify that the SWD and MMD markers fall outside the 3-sigma bands; if they do not, the metrics are not sensitive enough to detect the failure the paper claims they expose.","supporting_citations":[{"cited_title":"SN Computer Science 2(3), 210 (2021) https://doi.org/10","cited_arxiv_id":null,"evidence_quote":"Provides the BAT.jl Metropolis-Hastings sampler and DensitySampleVector interface used to generate the user samples under test."},{"cited_title":"Annual review of statistics and its application 6(1), 405–431 (2019)","cited_arxiv_id":null,"evidence_quote":"Gives the definition of the p-Wasserstein distance on which the sliced Wasserstein distance is based."},{"cited_title":"In: Scale Space and Variational Methods in Com- puter Vision: Third International Conference, SSVM 2011, Ein-Gedi, Israel, May 29–June 2, 2011, Revised Selected Papers 3, pp","cited_arxiv_id":null,"evidence_quote":"Introduces the random-projection sliced Wasserstein distance that the suite uses for high-dimensional comparisons."},{"cited_title":"The Journal of Machine Learning Research 13(1), 723–773 (2012)","cited_arxiv_id":null,"evidence_quote":"Defines maximum mean discrepancy in a reproducing kernel Hilbert space, including the kernel expansion the suite implements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Justifies the random Fourier features used to approximate the Gaussian-kernel MMD at large sample sizes."}],"review_version":1}