{"id":"fd609fef-5712-458a-8e90-375b0aacc5c1","arxiv_id":"2501.06022","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A comparative benchmark of six Markov Chain Monte Carlo and nested samplers finds that HMC and nested sampling perform best on complex cosmological parameter spaces, though no method dominates overall.","lead":"This paper benchmarks six Bayesian sampling methods on standard test distributions and on a dark-energy BAO likelihood. It finds that no single sampler dominates, with HMC and nested sampling handling complex parameter spaces better but at higher computational cost.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark lacks a common computational budget and controlled parallelism, so the claimed HMC runtime advantage may be a configuration artifact rather than an intrinsic sampler property.","rationale":"The reader's weakest assumption — that the benchmark enforces no common accuracy target or computational budget, making runtime and efficiency rankings potentially artifactual — is exactly the load-bearing concern for the paper's central comparative claims. The abstract and Section 5.2 promise guidance on which sampler to use, but the evidence for that guidance is a comparison where each method receives a different quantity of computation and where wall-clock time is not normalized for parallelization. If the benchmark were re-run under a common budget or with a fixed target ESS, the ranking could change substantially, and the practical recommendations would need revision. This is not a fatal flaw: the qualitative findings are plausible and consistent with prior sampler comparisons, and the test-problem results provide a useful sanity check. However, the lack of code release and the absence of error bars on the metrics compound the uncertainty, which is why the verdict should remain CONDITIONAL pending a controlled re-benchmark or clear disclosure of resource usage. The concern is addressable and does not invalidate the paper's broader message that no single method is universally superior, but it does mean the specific 'HMC is best' statement is not yet supported as stated.","tokens_in":13637,"tokens_out":8464,"duration_ms":85294,"concrete_test":"Re-run the wwaCDM (3D) and OmegaKwwaCDM (4D) cosmological benchmarks with a fixed likelihood-evaluation budget (e.g., 100,000 calls per sampler) and with every sampler executed serially on identical hardware, then compare the resulting parameter accuracy and effective sample size. If HMC no longer dominates, the runtime ranking is an artifact of the original per-sampler settings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 5.2) is that HMC provides the best balance of runtime and accuracy, especially in higher dimensions, and that no single sampler is significantly better overall (Section 6). This claim rests on a comparative benchmark in which each sampler is run with a fixed but arbitrary configuration (Appendix C): traditional MCMC uses 4 chains of 5000 draws with 50% burn-in, emcee uses 1000 steps with max(20*ndim,40) walkers, dynesty uses 1000 live points, PolyChord uses 100 live points, and HMC uses 500 warmup steps with an unspecified number of post-warmup draws. Crucially, no common likelihood-evaluation budget, target effective sample size, or wall-clock budget is enforced. Runtime is reported in wall-clock seconds, yet the samplers are not run under equal computational resources: §3 notes that dynesty, PolyChord, and emcee support parallelization via MPI or pools, while the HMC NumPyro implementation appears serial; §5.4 acknowledges the difficulty of measuring memory for parallel nested samplers but does not control core counts for runtime. Consequently, the headline comparison conflates algorithmic merit with implementation choices, sample-count settings, and parallelization, so the conclusion that HMC strikes the best runtime–accuracy balance could be an artifact of the test protocol rather than a robust property of the method. The paper's own caveat in §5.4 about implementation challenges does not resolve this because it does not impose a common budget or report CPU-time on equivalent resources.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper benchmarks six samplers—traditional Metropolis-Hastings MCMC via PyMC, HMC via NumPyro, a custom slice-sampling implementation, emcee, dynesty nested sampling, and PolyChord—on three synthetic distributions (correlated Gaussian, Rosenbrock, Gaussian mixture) and on a marginalized BAO likelihood with 1–4 cosmological parameters. Performance is assessed via runtime, memory usage, effective sample size per second, initial-seed sensitivity, and two normalized accuracy metrics. The paper concludes that HMC provides the best balance of runtime and accuracy, especially in higher dimensions, and that no single sampler is significantly better overall, with nested samplers recommended when evidence computation is required.","tokens_in":1,"tokens_out":4800,"duration_ms":97400,"significance":"A careful, reproducible comparison of widely used samplers on realistic cosmological likelihoods would be a valuable practical reference for the community. The paper's strengths are the use of standard test problems, a DESI-based BAO likelihood that is not trivial, and the intention to release the benchmark code. However, the central comparative claims are not currently supported by the evidence: the samplers are run under substantially different sample counts and live-point settings, accuracy normalization is arbitrary, and runtime and ESS metrics are reported without error bars or control for parallelization. If these issues are addressed, the paper could become a useful guide for sampler selection; in its present form, the conclusions are not robust.","major_comments":[{"comment":"The sampler configurations in Appendix C impose very different computational efforts: dynesty uses nlive=1000, PolyChord nlive=100, emcee uses 1000 steps, traditional MCMC uses 5000 draws, HMC uses 500 warmup steps with an unspecified number of post-warmup draws, and slice sampling uses width=0.1. No common accuracy target, effective-sample-size target, or likelihood-evaluation budget is enforced, so the runtime and ESS-per-second rankings in Figures 2 and 5 conflate algorithmic performance with configuration choices. This directly affects the central claim in Section 5.2 that HMC provides the best balance of runtime and accuracy; the benchmark should be rerun under a common budget, for example a fixed number of likelihood evaluations or a target ESS, or the choice of settings should be explicitly justified as a fair basis for comparison.","section":"Appendix C, Section 5.2"},{"comment":"Runtime, memory, and ESS per second are reported as single point values without error bars or repeated-run statistics. The 'Init sensitivity' metric tracks seed variation of parameter accuracy only, not variation of runtime or ESS. Given that the paper's main conclusions are about runtime and efficiency, the lack of variance estimates makes it impossible to judge whether differences such as the factor of ten between HMC and PolyChord in the 3D cosmological case are significant or within run-to-run noise.","section":"Section 3, Figure 5"},{"comment":"The normalized accuracy scores depend on arbitrarily fixed maximum error thresholds (0.5 for Gaussian, 2.0 for Rosenbrock, 5.0 for the mixture). Because these thresholds are not derived from any property of the target distributions, the absolute accuracy values in Figure 3 are meaningful only relative to the chosen normalization, and the rankings could change if different thresholds were adopted. The paper should either justify the thresholds or report the unnormalized errors alongside the normalized versions.","section":"Appendix A, Eqs. (A1)–(A4)"},{"comment":"The text states that 'the traditional method was convergent only for the Gaussian distribution,' yet Figure 3 reports traditional MCMC accuracy values for the Rosenbrock and mixture problems. If the chains did not converge, those accuracy values should be excluded or explicitly flagged as non-converged; as written, the figure contradicts the text and the comparison is unclear.","section":"Section 4.2, Figure 3"},{"comment":"The paper acknowledges in Section 5.4 that memory usage for parallel nested samplers is difficult to measure and may be underestimated, and Section 3 notes that dynesty, PolyChord, and emcee support parallelization while the HMC NumPyro implementation appears serial. No core counts or parallel settings are reported, so the runtime and memory comparisons mix serial and parallel execution. The benchmark should specify and control the number of cores or processes used for each sampler, or at least quantify the effect of parallelism on the reported metrics.","section":"Section 5.4, Section 3"}],"minor_comments":[{"comment":"The symbol Z is used both for the Bayesian evidence and for the normalization term in the marginalized BAO likelihood (Eq. A6); using a different symbol for one of these would avoid ambiguity.","section":"Section 2.4 and Appendix C, Eq. (A6)"},{"comment":"The description '5000 draws (ndraws ×5)' is ambiguous; it should be clarified whether 5000 is the total number of draws across all chains or the number per chain.","section":"Section 3"},{"comment":"The y-axis label 'Memory (MB)' includes values up to 70000; using a log scale or expressing the units in GB would improve readability.","section":"Figure 5"},{"comment":"There is a typo in 'developped' that should be corrected to 'developed'.","section":"Section 2.5"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be an MDPI-format preprint with header dates already showing 'Accepted'; if this is a preprint, the editorial headers should be removed to avoid confusion. The benchmark is less comprehensive than existing comparisons such as Albert et al. (2024, arXiv:2409.18464), which is cited but not directly compared, and the code is only promised 'upon publication,' which limits reproducibility. The scientific content is potentially salvageable, but the central claims require a rerun of the experiments under controlled computational budgets, replicated runs with error bars, and a defensible accuracy normalization. The editor may also wish to confirm that the journal's policy on posting the accepted version is consistent with the header dates."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a practical comparison of six samplers on standard test problems and a marginalized DESI BAO likelihood, with a credible qualitative takeaway: no single method is universally superior, and the choice should depend on problem geometry, gradient availability, and the need for evidence. That message is probably right, and it matches prior work including ref [8], so the paper's real value is as a cosmology-specific quantitative benchmark rather than as a source of new methodology.\n\nThe paper does several things well. It applies a consistent interface and metric set across very different samplers, which is genuinely annoying to do. It includes a Rosenbrock-style curved degeneracy and a multimodal test case, both relevant to cosmology. It also reports runtime, memory, ESS per second, initialization sensitivity, and two accuracy measures, and it is unusually frank about implementation difficulties in Section 5.4. The honest caveats earn credit.\n\nThe soft spots are real and, in one place, load-bearing. There is no common computational budget or controlled parallelism: dynesty, PolyChord, and emcee can run in parallel, while the HMC implementation appears serial, so the wall-clock runtime comparison that drives the 'HMC does best' claim in Section 5.2 could easily be an artifact. The sampler settings are also unequal (dynesty nlive=1000 vs PolyChord nlive=100), with no tuning to a common accuracy or ESS target. Metrics have no error bars, and the accuracy normalization thresholds (0.5, 2.0, 5.0) are arbitrary. The paper even contains an internal inconsistency: Section 4.2 says traditional MCMC was convergent only for the Gaussian test, yet traditional MCMC results are presented and used for the BAO cosmology runs. Memory measurements are acknowledged as unreliable for parallel nested samplers, and the ESS metric is not designed for nested output. None of this invalidates the central 'no universal winner' conclusion, but it means the quantitative rankings, especially the HMC runtime advantage, should be treated as provisional.\n\nWho is this for? Practitioners looking for a rough guide to sampler behavior on cosmological likelihoods will get something useful, especially once the code is actually released. The paper deserves a serious referee because the topic is important and the benchmark is reproducible in principle, but it needs revision: the authors should release code, enforce a common compute budget (e.g., run all samplers on the same cores with MPI disabled), report repeated-run scatter, and fix the traditional-MCMC convergence contradiction. I would not cite it in my own work as a definitive result, but I would cite it as an example of a community-visible sampler comparison on BAO data.","headline":"A useful but uneven sampler benchmark on a realistic BAO likelihood; the qualitative 'no universal winner' conclusion likely holds, but the headline HMC runtime advantage is not controlled enough to trust as a general statement.","tokens_in":14425,"tokens_out":2609,"would_cite":false,"duration_ms":28811,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"No single Bayesian sampler is universally superior; Hamiltonian Monte Carlo offers the best speed–accuracy trade-off for higher-dimensional cosmological inference, while nested samplers justify their higher cost by computing the evidence…","keywords":["cosmology","Markov chain Monte Carlo","Hamiltonian Monte Carlo","nested sampling","slice sampling","baryon acoustic oscillations","Bayesian evidence","sampler benchmarking"],"falsifier":"Rerun the same BAO benchmark with a matched accuracy target or a matched computational budget (for instance, requiring every sampler to reach the same effective sample size or the same evidence precision) and check whether HMC still has the best runtime while nested samplers remain the most expensive; if the rankings change under equalized settings, the reported ordering is an artifact of configuration.","tokens_in":1606,"feed_emoji":"🔭","tokens_out":4867,"duration_ms":80034,"temperature":0.7,"pith_summary":"The paper asks which Markov chain Monte Carlo sampler cosmologists should trust when estimating parameters from data such as baryon acoustic oscillations. It runs six samplers—a traditional random-walk MCMC, Hamiltonian Monte Carlo, slice sampling, an ensemble sampler, and two nested samplers—on identical test distributions and on a cosmological likelihood with one to four free parameters. Its central finding is that no single method is universally better: all recover well-constrained parameters reliably, and the ranking depends on dimensionality, curved degeneracies, and whether the Bayesian evidence is needed. When gradients are available, Hamiltonian Monte Carlo gives the best balance of runtime and accuracy in higher dimensions, while nested samplers remain accurate at higher computational cost and directly supply the evidence.","feed_headline":"No single sampler wins; HMC is best all-round for cosmology","feed_subtitle":"Six samplers, three toy distributions, and real BAO data: let problem geometry set the method.","key_machinery":"The load-bearing mechanism is the unified benchmark framework that standardizes metrics—runtime, memory usage, effective sample size per second, initialization sensitivity, and mean-based plus distribution-based accuracy—across all samplers on the same targets. Within that framework, the deciding quantities are the gradient-driven leapfrog dynamics of HMC, which preserve volume in phase space, and the live-point evidence integral of nested sampling, which converts the multi-dimensional evidence integral into a one-dimensional integration over prior mass. The test suite uses a correlated Gaussian, a Rosenbrock banana distribution, and a Gaussian mixture; the cosmological case uses a marginalized BAO likelihood with $\\Omega_m$, $\\Omega_K$, $w_0$, and $w_a$ as free parameters.","core_discovery":"The paper claims that sampler choice in cosmological inference should be driven by problem geometry and by what the analysis needs, because the tested samplers are comparable in accuracy yet differ markedly in cost. In the test problems, all samplers handle Gaussian targets, but curved degeneracies and multimodality separate them: traditional MCMC and slice sampling lose efficiency as dimension grows, HMC holds up on curved targets, and nested samplers stay reliable on multimodal ones. On the baryon-acoustic-oscillation likelihood, HMC has the best runtime and accuracy, especially for the 3- and 4-parameter dark-energy models; nested samplers cost more but remain accurate and compute the evidence; traditional MCMC and the ensemble sampler are adequate at low dimension but widen uncertainties in higher dimension. The paper concludes that no single method is significantly better across all settings, and that choosing a sampler means weighing dimensionality, likelihood features, and computational constraints.","pith_inferences":["A natural testable extension would fix a common effective-sample-size target or a common wall-clock budget across all samplers; the paper's rankings, especially HMC's runtime advantage, may depend on the chosen per-sampler settings such as 1000 versus 100 live points.","The paper's emphasis on geometry suggests that future comparisons should include non-differentiable or expensive likelihoods where gradients are unavailable, a condition that would erode HMC's main advantage.","If the no-single-method conclusion generalizes, cosmology pipelines should choose samplers by the downstream need—evidence versus posterior—and treat the two nested samplers and HMC as complementary tools rather than substitutes."],"forward_implications":["For well-constrained parameters like $\\Omega_m$, all samplers give consistent constraints, so the simplest methods are safe in low-dimensional problems.","Higher-dimensional dark-energy models should default to HMC when the likelihood is differentiable and the evidence is not required, since it delivers accuracy at a fraction of the runtime of nested samplers.","When Bayesian model comparison is the goal, nested samplers are the practical route because they output the evidence $Z$ directly, whereas MCMC evidence estimates are approximate.","Effective sample size per second alone can mislead: one sampler shows high ESS but larger parameter uncertainties in higher dimensions, so raw sampling efficiency does not guarantee optimal posterior exploration.","Relying on a single sampler family risks geometry-induced bias; the observed differences in curved and multimodal targets imply that multiple sampling approaches are worth running for non-standard parameter spaces."],"supporting_citations":[{"why":"Provides the original acceptance rule that defines the traditional MCMC baseline.","marker":"[4]"},{"why":"Supplies the general Metropolis–Hastings criterion used by the traditional sampler.","marker":"[5]"},{"why":"Introduces Hamiltonian Monte Carlo, whose leapfrog dynamics are the basis of the HMC implementation.","marker":"[9]"},{"why":"Defines slice sampling and the stepping-out procedure used in the custom implementation.","marker":"[10]"},{"why":"Introduces nested sampling and the evidence integral that the nested samplers compute.","marker":"[11]"},{"why":"Introduces the PolyChord algorithm and its slice-sampling proposal mechanism used in the benchmark.","marker":"[12,13]"},{"why":"Provides the ensemble sampler used as one of the six compared methods.","marker":"[17]"},{"why":"Supplies the dynamic nested sampling implementation and its evidence calculation used by the nested sampler.","marker":"[18,19]"},{"why":"Supplies the BAO measurements that the cosmological likelihood is fitted to.","marker":"[25]"},{"why":"Derive and validate the marginalized BAO likelihood that forms the cosmological test case.","marker":"[26,27]"}],"fun_headline_variants":["HMC best for BAO cosmology, nested for evidence, no single winner","Sampler choice hinges on geometry: HMC for curved, nested for multimodal","No one sampler rules, but HMC shines on cosmological data","Rethink MCMC choice: geometry and cost trump sampling accuracy","Cosmology MCMC face-off: HMC leads, but nested computes evidence"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"The comparison assumes that the fixed per-sampler settings—such as the number of live points, walkers, warmup steps, and burn-in fractions—place the methods on an equal footing, rather than merely reflecting arbitrary configuration choices.","fun_headline_variants_meta":{"raw":{"variants":["HMC best for BAO cosmology, nested for evidence, no single winner","Sampler choice hinges on geometry: HMC for curved, nested for multimodal","No one sampler rules, but HMC shines on cosmological data","Rethink MCMC choice: geometry and cost trump sampling accuracy","Cosmology MCMC face-off: HMC leads, but nested computes evidence"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1339,"prompt_tokens":882,"completion_tokens":457,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":498,"completion_tokens_details":{"reasoning_tokens":360}},"tokens_in":498,"tokens_out":457,"duration_ms":4983,"temperature":1.0,"reasoning_tokens":360,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:05:34.855108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the same BAO benchmark with a matched accuracy target or a matched computational budget (for instance, requiring every sampler to reach the same effective sample size or the same evidence precision) and check whether HMC still has the best runtime while nested samplers remain the most expensive; if the rankings change under equalized settings, the reported ordering is an artifact of configuration.","supporting_citations":[{"cited_title":"Equation of state calculations by fast computing machines","cited_arxiv_id":null,"evidence_quote":"Provides the original acceptance rule that defines the traditional MCMC baseline."},{"cited_title":"Monte Carlo sampling methods using Markov chains and their applications","cited_arxiv_id":null,"evidence_quote":"Supplies the general Metropolis–Hastings criterion used by the traditional sampler."},{"cited_title":"Hybrid Monte Carlo","cited_arxiv_id":null,"evidence_quote":"Introduces Hamiltonian Monte Carlo, whose leapfrog dynamics are the basis of the HMC implementation."},{"cited_title":"Slice sampling","cited_arxiv_id":null,"evidence_quote":"Defines slice sampling and the stepping-out procedure used in the custom implementation."},{"cited_title":"Nested sampling for general Bayesian computation","cited_arxiv_id":null,"evidence_quote":"Introduces nested sampling and the evidence integral that the nested samplers compute."}],"review_version":1}