REVIEW 4 major objections 5 minor 1 cited by
Automated calibration of simulated galaxy catalogues for cosmological analyses
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read An automated pipeline calibrates 23-parameter galaxy mocks against SDSS observations.
desk verdict A genuinely useful automated mock-calibration pipeline with an honest but incomplete single-seed demonstration. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the differential evolution algorithm, a population-based stochastic minimiser in which candidate parameter vectors are mutated by scaled differences of randomly chosen population members and recombined into a trial population that replaces worse candidates. This is used to minimise the total objective $\chi^2 = \chi^2_{\rm LF} + \chi^2_{\rm GC} + \chi^2_{\rm colour}$, comparing mock and SDSS measurements of the luminosity function, the projected two-point correlation function by luminosity threshold, and the same correlation split into blue and red galaxies. The galaxy recipe being tuned is the halo occupation distribution plus sub-halo abundance matching model with 23 free parameters controlling satellite occupation, abundance-matching scatter, satellite luminosities, and satellite colour fractions.
What would settle it
Estimate the full covariance of all observables from an ensemble of mock realisations and re-run the differential evolution minimisation using that covariance instead of the diagonal approximation; if the best-fit 23-parameter vector shifts by more than the run-to-run scatter, the current objective is biasing the calibration.
Extended reading notes
Core claim
The central claim is that calibrating a high-dimensional, stochastic mock generator can be made fully automatic by treating the problem as a noisy optimisation and searching directly for the best-fit parameter vector rather than sampling the posterior. With a 23-parameter HOD+SHAM recipe, the differential-evolution search converges after about 600 function evaluations per cosmology and yields projected correlation functions in good agreement with SDSS for all luminosity thresholds considered, with the closest agreement below roughly $1\,h^{-1}\mathrm{Mpc}$ inside the one-halo term. The same pipeline calibrates mocks built on both general-relativity and $f(R)$ halo catalogues, and the best-fit galaxy-halo parameters differ between the two cosmologies in a way that re-absorbs the difference in gravity model at low redshift.
Load-bearing premise
The load-bearing assumption is that the chi-squared objective can treat every luminosity bin, separation bin, and the three observables as independent, with an ad hoc uncertainty of $10^{-3}$ on the luminosity function; if the real covariances are strongly non-diagonal, the best-fit parameters could be biased and the reported $\chi^2$ values would not be formally meaningful.
Editorial extensions
If this is right
- The same automated calibration can be applied to any parametric galaxy-assignment recipe, not only the HOD+SHAM model demonstrated here.
- Because each calibration converged in around 600 evaluations and roughly 36 hours of wall-clock time for two ~5-million-galaxy catalogues, recalibrating mocks as survey data improve becomes practical.
- The modified-gravity mock reproduces the same SDSS observations as the standard mock, so differences between the two gravity models at $z=0.1$ can be absorbed by the galaxy-halo connection parameters.
- The calibrated modified-gravity catalogue provides a realistic testbed for forecasting how next-generation galaxy surveys might constrain gravity models.
- The convergence criterion and hyper-parameter choices transfer to larger parameter spaces, so the method is a candidate for automating the calibration of future massive mocks.
Reading between the lines
- We infer that replacing the diagonal, ad hoc noise model with a jackknife covariance estimated from the mocks themselves is a direct next step; the pipeline already accepts a covariance matrix and would then yield formally interpretable $\chi^2$ values.
- We infer the method could also calibrate emulator-based forward models, where each evaluation is cheap, enabling far more evaluations and a fuller exploration of the 23-dimensional landscape.
- We infer that running the calibration from several independent initialisations would turn run-to-run scatter in the best-fit vector into a practical estimate of calibration uncertainty, which the paper does not quantify.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an automated calibration pipeline for galaxy mocks based on the differential evolution stochastic minimisation algorithm. The pipeline is applied to halo catalogues from both LambdaCDM and f(R) modified gravity simulations, populating haloes with galaxies using a combined HOD+SHAM recipe with 23 free parameters. The objective function compares mock measurements to SDSS observations of the luminosity function, the projected two-point correlation function for several luminosity thresholds, and the colour-split projected correlation function. The pipeline converges in both cosmologies, yielding mocks that reproduce the 1-halo clustering well, with some underprediction of the clustering of faint red galaxies. The authors also present a qualitative comparison of the calibrated parameters between the two gravity models, while explicitly declining to interpret the differences as significant.
Significance. If the calibration is robust, the pipeline would be a valuable tool for future surveys (Euclid, DESI, LSST) that require large mock catalogues with many free parameters. The paper demonstrates the feasibility of stochastic optimisation in a 23-dimensional parameter space with a realistic galaxy-halo connection recipe and shows that the approach is, in principle, agnostic to the underlying gravity model. The authors are commendably transparent about limitations: they state that parameter uncertainties are not provided, that the cross-cosmology comparison is only qualitative, and that the model lacks ingredients to adjust colour- and luminosity-dependent clustering. The main shortcoming is that the validation is entirely in-sample and rests on a single stochastic optimisation run per cosmology; without repeated seeds or resampling, the reported 'properly calibrated' claim is not yet distinguished from a favourable noise draw. The central idea is sound and the gaps are fixable, but the evidence as presented is not yet sufficient to support the strength of the abstract's claim.
major comments (4)
- [Section 3, Eq. (20)] The stopping criterion is the standard deviation of the chi^2 values of the population falling below a tolerance, but each chi^2 evaluation is computed from a freshly generated random mock. The population spread may therefore be dominated by Monte Carlo realisation noise rather than by the distance to the true minimum, so convergence under Eq. (20) does not guarantee that a genuine minimum has been located. To support the claim that the pipeline 'properly calibrates', the authors should run the optimisation with multiple random seeds and/or generate several independent mocks at the nominal best-fit parameters, showing that the chi^2 and the inferred parameters are stable under re-sampling.
- [Section 5, Eqs. (21)-(24)] The total chi^2 treats all luminosity bins, separation bins, and the three observables (luminosity function, total clustering, colour-split clustering) as statistically independent, and the value sigma_LF = 10^-3 is set ad hoc. As a consequence, the reported chi^2 per degree-of-freedom values are not formally interpretable as goodness-of-fit measures, and the best fit could be biased if the true covariance is significantly non-diagonal. The paper should either include an approximate covariance (e.g., from jackknife or mock realisations) when available, or explicitly state that Eq. (24) is a weighted loss function and avoid presenting reduced chi^2 values as quantitative goodness-of-fit indicators.
- [Section 5, Figs. 3 and 5] The clustering of red galaxies in the faint luminosity bins is systematically lower in the mocks than in the observations, and the authors attribute this to the absence of assembly bias, conformity, and colour segregation in the recipe. This is a model limitation that directly affects the claim of successful calibration for colour-dependent statistics. The authors should quantify the discrepancy (e.g., the contribution of these bins to the total chi^2) and discuss the practical implications for analyses that use the colour-dependent clustering of these mocks.
- [Section 5, Figs. 2-5] The validation is in-sample: the figures that demonstrate agreement with observations correspond to the very objective that was minimised, and the final mock is generated with the same parameters and the same set of observables. This does not constitute an out-of-sample test of the pipeline's predictive power. A stronger demonstration would hold out one observable (e.g., the luminosity function or a specific luminosity bin) or one scale range during calibration and check the agreement on that held-out quantity, or compare against an observable not included in the objective, such as redshift-space clustering or galaxy-galaxy lensing.
minor comments (5)
- [Section 3 and footnote 1] The term 'Latyn hyper-cube' appears several times; this should be 'Latin hypercube'.
- [Eq. (12)] There appears to be a typographical error in the first line of Eq. (12): the factor of 2 should evidently multiply (v_max/(H0 r_max))^2, and the square is missing in the typeset expression.
- [Section 4 and Eqs. (22)-(23)] The text mentions that the custom correlation function implementation integrates jackknife resampling for sample variance, but it is not stated how the resulting uncertainty enters the sigma_wmock_p terms in Eqs. (22) and (23); please clarify this connection.
- [Section 5, Figs. 2-5] The terms '1-halo term' and '2-halo term' are used without definition; consider defining them at first use in Section 5.
- [Fig. 1 caption] The caption reads 'Histograms of the chi2 value as a function of the number of evaluations', but the figure is a two-dimensional histogram; please rephrase to describe the colour map and binning more accurately.
Circularity Check
Explicit calibration exercise with in-sample agreement; no prediction is dressed up as an independent derivation.
full rationale
The paper is an explicit fitting/calibration exercise, not a derivation of predictions from first principles. Equation (24) defines the objective that the differential evolution algorithm minimizes, and Figs. 2-5 show agreement on the same observables that entered Eq. (24); this is in-sample validation, which is appropriate for a calibration pipeline and is not presented as an out-of-sample prediction. The SHAM equality in Eq. (8) does make the cumulative luminosity function match the observed input by construction, but the authors do not use luminosity-function agreement as an independent success claim; the headline validation is the projected correlation function, which is not fixed by Eq. (8). Self-citations (Carretero et al. 2015, Arnold et al. 2019, Euclid Collaboration: Castander et al. 2025) supply the mock recipe and halo catalogues used as inputs; they are not invoked to guarantee the calibration result, and no uniqueness theorem or ansatz is smuggled in via self-citation. The acknowledged lack of parameter uncertainties and the stochastic noise visible in Fig. 1 are robustness concerns, not circularity.
Assumptions & free parameters
free parameters (7)
- HOD parameters alpha, a1, a2, a3, b1, b2, s1, s2 =
not reported numerically (see Fig. 6)
- SHAM scatter sigma_logL and luminosity threshold L_AM =
not reported numerically
- Satellite luminosity relation parameters a_AM, b_AM, c_AM =
not reported numerically (see Fig. 7)
- Colour fractions f_red_sat,i and f_green_sat,i for i=1..5 =
not reported numerically (see Fig. 8)
- DE hyperparameters: mutation M in [0.5,1], recombination R=0.7, population size 23, tolerances A=0.0 and T=0.1 =
set by hand as described in Sect. 3
- Luminosity function error sigma_LF =
10^-3
- Satellite-to-central luminosity cap L_sat <= 1.05 L_cen =
1.05
assumptions (6)
- domain assumption The halo catalogues from Arnold et al. (2019) accurately represent the dark matter distribution for both LambdaCDM and f(R) gravity at the resolution used (M_part = 4.5e9 h^-1 M_sun).
- domain assumption The assumed NFW profile with a spherical shape and the concentration relation from Springel et al. (2008) (Eq. 12) describe the satellite distribution within haloes.
- domain assumption The SHAM relation equating cumulative number densities (Eq. 8) is a valid mapping between halo mass and galaxy luminosity.
- domain assumption The SDSS measurements of the luminosity function, projected correlation function, and colour distributions at z about 0.1 are the correct targets for calibration.
- domain assumption The Poisson distribution for satellite counts (Eq. 2) and the four-parameter Schechter function for satellite luminosities are adequate models.
- domain assumption The f(R) modified gravity model with |f_R0| = 10^-5 and its simulation via MG-Gadget are reliable.
Cite this review
Pith. "Pith review of Automated calibration of simulated galaxy catalogues for cosmological analyses." pith.science (2026). https://pith.science/paper/AXR2DLIZ
@misc{pith2026250723451,
author = {Pith},
title = {Pith review of: Automated calibration of simulated galaxy catalogues for cosmological analyses},
year = {2026},
howpublished = {\url{https://pith.science/paper/AXR2DLIZ}},
note = {Machine review of arXiv:2507.23451}
}
abstract
Simulated galaxy catalogues have become an essential tool for preparing and exploiting observations from galaxy surveys. They constitute a key ingredient in modelling the systematic uncertainties present in the analysis. However, in order to reach the large volume and high precision required for galaxy surveys, we generally populate dark matter haloes with galaxies following certain theoretical recipes. Such recipes contain free parameters that are calibrated comparing the simulations against observations, but the creation of galaxy mocks is a stochastic process with a large number of free parameters to calibrate. We present a new pipeline, based on the differential evolution algorithm, that can calibrate galaxy mocks in a fully automated way for realistic scenarios with a large parameter space. We apply the pipeline to galaxy mocks built on a combination of halo occupation distribution and sub-halo abundance matching techniques. We show that our pipeline can properly calibrate the galaxy mocks against observations for both $\Lambda$CDM and modified gravity halo catalogues. This type of calibration pipeline provides a new tool for automating the calibration of future massive galaxy mocks.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Euclid. Populating a dark universe with galaxies using SciPIC
SciPICal calibrates galaxy properties in halo-based mocks for Euclid, delivering improved clustering predictions and a deep catalogue validated to 15 percent against observations.
Reference graph
Works this paper leans on
-
[1]
Abbott, T. M. C., Aguena, M., Alarcon, A., et al. 2022, PRD, 105, 023520
work page 2022
-
[2]
G., Aguilar, J., Ahlen, S., et al
Adame, A. G., Aguilar, J., Ahlen, S., et al. 2025, JCAP, 02, 021
2025
-
[3]
Alam, S., de Mattia, A., Tamone, A., et al. 2021, MNRAS, 504, 4667
work page 2021
-
[4]
Arnold, C., Fosalba, P., Springel, V ., Puchwein, E., & Blot, L. 2019, MNRAS, 483, 790
work page 2019
-
[5]
Benson, A. J. 2012, New A, 17, 175
work page 2012
-
[6]
R., Lupton, R
Blanton, M. R., Lupton, R. H., Schlegel, D. J., et al. 2005, ApJ, 631, 208
2005
-
[7]
R., Schlegel, D
Blanton, M. R., Schlegel, D. J., Strauss, M. A., et al. 2005, AJ, 129, 2562
2005
-
[8]
Buchdahl, H. A. 1970, MNRAS, 150, 1
work page 1970
Show all 30 references
-
[9]
J., Gaztañaga, E., Crocce, M., & Fosalba, P
Carretero, J., Castander, F. J., Gaztañaga, E., Crocce, M., & Fosalba, P. 2015, MNRAS, 447, 646
2015
-
[10]
2017, in Proceedings of the European Physical Society Conference on High Energy Physics
Carretero, J., Tallada, P., Casals, J., et al. 2017, in Proceedings of the European Physical Society Conference on High Energy Physics. 5-12 July, 488
2017
-
[11]
J., Gaztañaga, E., Fosalba, P., & Carretero, J
Crocce, M., Castander, F. J., Gaztañaga, E., Fosalba, P., & Carretero, J. 2015, MNRAS, 453, 1513
2015
-
[12]
H., Becker, M
DeRose, J., Wechsler, R. H., Becker, M. R., et al. 2022, PRD, 105, 123520 Euclid Collaboration: Castander, F. J., Fosalba, P., Stadel, J., et al. 2025, A&A, 697, A5 Euclid Collaboration: Knabenhans, M., Stadel, J., Potter, D., et al. 2021, MN- RAS, 505, 2840
2022
-
[13]
2021, A&A, 656, A106
Ferrero, I., Crocce, M., Tutusaus, I., et al. 2021, A&A, 656, A106
2021
-
[14]
2021, A&A, 646, A140
Heymans, C., Tröster, T., Asgari, M., et al. 2021, A&A, 646, A140
2021
-
[15]
& Sawicki, I
Hu, W. & Sawicki, I. 2007, PRD, 76, 064004
2007
-
[16]
& Roos, M
James, F. & Roos, M. 1975, Comput. Phys. Commun., 10, 343
1975
-
[17]
2001, SciPy: Open source scientific tools for Python
Jones, E., Oliphant, T., Peterson, P., et al. 2001, SciPy: Open source scientific tools for Python
2001
-
[18]
F., Frenk, C
Navarro, J. F., Frenk, C. S., & White, S. D. M. 1997, ApJ, 490, 493
1997
-
[19]
2013, MNRAS, 436, 348
Puchwein, E., Baldi, M., & Springel, V . 2013, MNRAS, 436, 348
2013
-
[20]
2020, MNRAS, 499, 269
Smith, A., Burtin, E., Hou, J., et al. 2020, MNRAS, 499, 269
2020
-
[21]
2005, MNRAS, 364, 1105
Springel, V . 2005, MNRAS, 364, 1105
2005
-
[22]
2010, ARA&A, 48, 391
Springel, V . 2010, ARA&A, 48, 391
2010
-
[23]
2008, MNRAS, 391, 1685
Springel, V ., Wang, J., V ogelsberger, M., et al. 2008, MNRAS, 391, 1685
2008
-
[24]
& Price, K
Storn, R. & Price, K. 1997, J. Glob. Optim., 11, 341
1997
-
[25]
2020, A&C, 32, 100391
Tallada, P., Carretero, J., Casals, J., et al. 2020, A&C, 32, 100391
2020
-
[26]
2025, arXiv e-prints, arXiv:2504.04961
Viglione, C., Fosalba, P., Tutusaus, I., et al. 2025, arXiv e-prints, arXiv:2504.04961
2025 arXiv
-
[27]
F., Berlind, A
Watson, D. F., Berlind, A. A., McBride, C. K., Hogg, D. W., & Jiang, T. 2012, ApJ, 749, 83
2012
-
[28]
Wechsler, R. H. & Tinker, J. L. 2018, ARA&A, 56, 435
2018
-
[29]
M., & Bowen, D
Wormington, M., Panaccione, C., Matney, K. M., & Bowen, D. K. 1999, Phil. Trans. R. Soc. A, 357, 2827
1999
-
[30]
H., et al
Zehavi, I., Zheng, Z., Weinberg, D. H., et al. 2011, ApJ, 736, 59 Article number, page 10 of 10
2011
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.