Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Automated calibration of simulated galaxy catalogues for cosmological analyses

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read An automated pipeline calibrates 23-parameter galaxy mocks against SDSS observations.

desk verdict A genuinely useful automated mock-calibration pipeline with an honest but incomplete single-seed demonstration. read the letter →

arxiv 2507.23451 v1 pith:AXR2DLIZ submitted 2025-07-31 astro-ph.CO

classification astro-ph.CO
keywords galaxymockshalooccupationdistributionsub-haloabundancematchingdifferentialevolutionprojectedcorrelationfunctionluminositymodifiedgravityautomatedcalibration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that the free parameters linking dark-matter haloes to galaxies in simulated catalogues can be set automatically, without hand-tuning, even when the calibration space has 23 dimensions and the mock-generation process is stochastic. The authors build a pipeline that treats each galaxy mock as a noisy function evaluation and minimises a $\chi^2$ difference against SDSS measurements of the luminosity function, projected clustering, and colour-split clustering using the differential evolution algorithm. They show the calibration converges and reproduces the observations, particularly on small one-halo scales, for both a $\Lambda$CDM halo catalogue and an $f(R)$ modified-gravity catalogue. If correct, this removes a major bottleneck in building the large, high-precision mocks that galaxy surveys need for systematic-error modelling.

What carries the argument

The load-bearing mechanism is the differential evolution algorithm, a population-based stochastic minimiser in which candidate parameter vectors are mutated by scaled differences of randomly chosen population members and recombined into a trial population that replaces worse candidates. This is used to minimise the total objective $\chi^2 = \chi^2_{\rm LF} + \chi^2_{\rm GC} + \chi^2_{\rm colour}$, comparing mock and SDSS measurements of the luminosity function, the projected two-point correlation function by luminosity threshold, and the same correlation split into blue and red galaxies. The galaxy recipe being tuned is the halo occupation distribution plus sub-halo abundance matching model with 23 free parameters controlling satellite occupation, abundance-matching scatter, satellite luminosities, and satellite colour fractions.

What would settle it

Estimate the full covariance of all observables from an ensemble of mock realisations and re-run the differential evolution minimisation using that covariance instead of the diagonal approximation; if the best-fit 23-parameter vector shifts by more than the run-to-run scatter, the current objective is biasing the calibration.

Watch

Extended reading notes

Core claim

The central claim is that calibrating a high-dimensional, stochastic mock generator can be made fully automatic by treating the problem as a noisy optimisation and searching directly for the best-fit parameter vector rather than sampling the posterior. With a 23-parameter HOD+SHAM recipe, the differential-evolution search converges after about 600 function evaluations per cosmology and yields projected correlation functions in good agreement with SDSS for all luminosity thresholds considered, with the closest agreement below roughly $1\,h^{-1}\mathrm{Mpc}$ inside the one-halo term. The same pipeline calibrates mocks built on both general-relativity and $f(R)$ halo catalogues, and the best-fit galaxy-halo parameters differ between the two cosmologies in a way that re-absorbs the difference in gravity model at low redshift.

Load-bearing premise

The load-bearing assumption is that the chi-squared objective can treat every luminosity bin, separation bin, and the three observables as independent, with an ad hoc uncertainty of $10^{-3}$ on the luminosity function; if the real covariances are strongly non-diagonal, the best-fit parameters could be biased and the reported $\chi^2$ values would not be formally meaningful.

Editorial extensions

If this is right

  • The same automated calibration can be applied to any parametric galaxy-assignment recipe, not only the HOD+SHAM model demonstrated here.
  • Because each calibration converged in around 600 evaluations and roughly 36 hours of wall-clock time for two ~5-million-galaxy catalogues, recalibrating mocks as survey data improve becomes practical.
  • The modified-gravity mock reproduces the same SDSS observations as the standard mock, so differences between the two gravity models at $z=0.1$ can be absorbed by the galaxy-halo connection parameters.
  • The calibrated modified-gravity catalogue provides a realistic testbed for forecasting how next-generation galaxy surveys might constrain gravity models.
  • The convergence criterion and hyper-parameter choices transfer to larger parameter spaces, so the method is a candidate for automating the calibration of future massive mocks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • We infer that replacing the diagonal, ad hoc noise model with a jackknife covariance estimated from the mocks themselves is a direct next step; the pipeline already accepts a covariance matrix and would then yield formally interpretable $\chi^2$ values.
  • We infer the method could also calibrate emulator-based forward models, where each evaluation is cheap, enabling far more evaluations and a fuller exploration of the 23-dimensional landscape.
  • We infer that running the calibration from several independent initialisations would turn run-to-run scatter in the best-fit vector into a practical estimate of calibration uncertainty, which the paper does not quantify.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper presents an automated calibration pipeline for galaxy mocks based on the differential evolution stochastic minimisation algorithm. The pipeline is applied to halo catalogues from both LambdaCDM and f(R) modified gravity simulations, populating haloes with galaxies using a combined HOD+SHAM recipe with 23 free parameters. The objective function compares mock measurements to SDSS observations of the luminosity function, the projected two-point correlation function for several luminosity thresholds, and the colour-split projected correlation function. The pipeline converges in both cosmologies, yielding mocks that reproduce the 1-halo clustering well, with some underprediction of the clustering of faint red galaxies. The authors also present a qualitative comparison of the calibrated parameters between the two gravity models, while explicitly declining to interpret the differences as significant.

Significance. If the calibration is robust, the pipeline would be a valuable tool for future surveys (Euclid, DESI, LSST) that require large mock catalogues with many free parameters. The paper demonstrates the feasibility of stochastic optimisation in a 23-dimensional parameter space with a realistic galaxy-halo connection recipe and shows that the approach is, in principle, agnostic to the underlying gravity model. The authors are commendably transparent about limitations: they state that parameter uncertainties are not provided, that the cross-cosmology comparison is only qualitative, and that the model lacks ingredients to adjust colour- and luminosity-dependent clustering. The main shortcoming is that the validation is entirely in-sample and rests on a single stochastic optimisation run per cosmology; without repeated seeds or resampling, the reported 'properly calibrated' claim is not yet distinguished from a favourable noise draw. The central idea is sound and the gaps are fixable, but the evidence as presented is not yet sufficient to support the strength of the abstract's claim.

major comments (4)
  1. [Section 3, Eq. (20)] The stopping criterion is the standard deviation of the chi^2 values of the population falling below a tolerance, but each chi^2 evaluation is computed from a freshly generated random mock. The population spread may therefore be dominated by Monte Carlo realisation noise rather than by the distance to the true minimum, so convergence under Eq. (20) does not guarantee that a genuine minimum has been located. To support the claim that the pipeline 'properly calibrates', the authors should run the optimisation with multiple random seeds and/or generate several independent mocks at the nominal best-fit parameters, showing that the chi^2 and the inferred parameters are stable under re-sampling.
  2. [Section 5, Eqs. (21)-(24)] The total chi^2 treats all luminosity bins, separation bins, and the three observables (luminosity function, total clustering, colour-split clustering) as statistically independent, and the value sigma_LF = 10^-3 is set ad hoc. As a consequence, the reported chi^2 per degree-of-freedom values are not formally interpretable as goodness-of-fit measures, and the best fit could be biased if the true covariance is significantly non-diagonal. The paper should either include an approximate covariance (e.g., from jackknife or mock realisations) when available, or explicitly state that Eq. (24) is a weighted loss function and avoid presenting reduced chi^2 values as quantitative goodness-of-fit indicators.
  3. [Section 5, Figs. 3 and 5] The clustering of red galaxies in the faint luminosity bins is systematically lower in the mocks than in the observations, and the authors attribute this to the absence of assembly bias, conformity, and colour segregation in the recipe. This is a model limitation that directly affects the claim of successful calibration for colour-dependent statistics. The authors should quantify the discrepancy (e.g., the contribution of these bins to the total chi^2) and discuss the practical implications for analyses that use the colour-dependent clustering of these mocks.
  4. [Section 5, Figs. 2-5] The validation is in-sample: the figures that demonstrate agreement with observations correspond to the very objective that was minimised, and the final mock is generated with the same parameters and the same set of observables. This does not constitute an out-of-sample test of the pipeline's predictive power. A stronger demonstration would hold out one observable (e.g., the luminosity function or a specific luminosity bin) or one scale range during calibration and check the agreement on that held-out quantity, or compare against an observable not included in the objective, such as redshift-space clustering or galaxy-galaxy lensing.
minor comments (5)
  1. [Section 3 and footnote 1] The term 'Latyn hyper-cube' appears several times; this should be 'Latin hypercube'.
  2. [Eq. (12)] There appears to be a typographical error in the first line of Eq. (12): the factor of 2 should evidently multiply (v_max/(H0 r_max))^2, and the square is missing in the typeset expression.
  3. [Section 4 and Eqs. (22)-(23)] The text mentions that the custom correlation function implementation integrates jackknife resampling for sample variance, but it is not stated how the resulting uncertainty enters the sigma_wmock_p terms in Eqs. (22) and (23); please clarify this connection.
  4. [Section 5, Figs. 2-5] The terms '1-halo term' and '2-halo term' are used without definition; consider defining them at first use in Section 5.
  5. [Fig. 1 caption] The caption reads 'Histograms of the chi2 value as a function of the number of evaluations', but the figure is a two-dimensional histogram; please rephrase to describe the colour map and binning more accurately.

Circularity Check

0 steps flagged · score 0.0 of 10

Explicit calibration exercise with in-sample agreement; no prediction is dressed up as an independent derivation.

full rationale

The paper is an explicit fitting/calibration exercise, not a derivation of predictions from first principles. Equation (24) defines the objective that the differential evolution algorithm minimizes, and Figs. 2-5 show agreement on the same observables that entered Eq. (24); this is in-sample validation, which is appropriate for a calibration pipeline and is not presented as an out-of-sample prediction. The SHAM equality in Eq. (8) does make the cumulative luminosity function match the observed input by construction, but the authors do not use luminosity-function agreement as an independent success claim; the headline validation is the projected correlation function, which is not fixed by Eq. (8). Self-citations (Carretero et al. 2015, Arnold et al. 2019, Euclid Collaboration: Castander et al. 2025) supply the mock recipe and halo catalogues used as inputs; they are not invoked to guarantee the calibration result, and no uniqueness theorem or ansatz is smuggled in via self-citation. The acknowledged lack of parameter uncertainties and the stochastic noise visible in Fig. 1 are robustness concerns, not circularity.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claim rests on the 23 fitted galaxy-halo parameters (listed in Table 1), the hand-set DE hyperparameters, and a set of standard modeling assumptions about halo profiles, abundance matching, and the fidelity of the input N-body halo catalogues. No new physical entities are introduced.

free parameters (7)
  • HOD parameters alpha, a1, a2, a3, b1, b2, s1, s2 = not reported numerically (see Fig. 6)
    Control the mean satellite occupation and the M1/Mmin scaling in Eqs. 2-3; calibrated against SDSS luminosity function and clustering.
  • SHAM scatter sigma_logL and luminosity threshold L_AM = not reported numerically
    Scatter in the halo mass-galaxy luminosity relation and the luminosity above which scatter is applied; calibrated against SDSS data.
  • Satellite luminosity relation parameters a_AM, b_AM, c_AM = not reported numerically (see Fig. 7)
    Set L* as a function of L_cen in Eq. 10; calibrated against SDSS data.
  • Colour fractions f_red_sat,i and f_green_sat,i for i=1..5 = not reported numerically (see Fig. 8)
    10 free parameters controlling red/green satellite fractions in 5 luminosity bins; calibrated against SDSS colour-dependent clustering.
  • DE hyperparameters: mutation M in [0.5,1], recombination R=0.7, population size 23, tolerances A=0.0 and T=0.1 = set by hand as described in Sect. 3
    Chosen to balance convergence speed and global search; not calibrated automatically.
  • Luminosity function error sigma_LF = 10^-3
    Ad hoc uncertainty assigned to the SDSS luminosity function at high luminosities in Eq. 21; the paper states small variations have negligible impact.
  • Satellite-to-central luminosity cap L_sat <= 1.05 L_cen = 1.05
    Arbitrary cap to prevent satellites brighter than centrals; the paper says it is enough for proper LF calibration.
assumptions (6)
  • domain assumption The halo catalogues from Arnold et al. (2019) accurately represent the dark matter distribution for both LambdaCDM and f(R) gravity at the resolution used (M_part = 4.5e9 h^-1 M_sun).
    The calibration results depend on the input halo masses and concentrations; errors in the halo catalogues propagate into the best-fit parameters. Invoked in Sect. 5.
  • domain assumption The assumed NFW profile with a spherical shape and the concentration relation from Springel et al. (2008) (Eq. 12) describe the satellite distribution within haloes.
    Used in Eq. 11 to place satellites; the paper notes spherical NFW is a simplification and triaxial NFW was used in MICE.
  • domain assumption The SHAM relation equating cumulative number densities (Eq. 8) is a valid mapping between halo mass and galaxy luminosity.
    Core of the abundance matching technique; assumes a monotonic relation after applying scatter.
  • domain assumption The SDSS measurements of the luminosity function, projected correlation function, and colour distributions at z about 0.1 are the correct targets for calibration.
    The pipeline calibrates to these observations; if the SDSS data are biased, the calibrated mocks inherit the bias.
  • domain assumption The Poisson distribution for satellite counts (Eq. 2) and the four-parameter Schechter function for satellite luminosities are adequate models.
    Standard choices from Carretero et al. (2015); the paper does not test alternatives.
  • domain assumption The f(R) modified gravity model with |f_R0| = 10^-5 and its simulation via MG-Gadget are reliable.
    The modified gravity mock is built on these halo catalogues; using a different f(R) parameter or simulation code could change results.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automated calibration of simulated galaxy catalogues for cosmological analyses." pith.science (2026). https://pith.science/paper/AXR2DLIZ

@misc{pith2026250723451,
  author       = {Pith},
  title        = {Pith review of: Automated calibration of simulated galaxy catalogues for cosmological analyses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AXR2DLIZ}},
  note         = {Machine review of arXiv:2507.23451}
}
abstract

Simulated galaxy catalogues have become an essential tool for preparing and exploiting observations from galaxy surveys. They constitute a key ingredient in modelling the systematic uncertainties present in the analysis. However, in order to reach the large volume and high precision required for galaxy surveys, we generally populate dark matter haloes with galaxies following certain theoretical recipes. Such recipes contain free parameters that are calibrated comparing the simulations against observations, but the creation of galaxy mocks is a stochastic process with a large number of free parameters to calibrate. We present a new pipeline, based on the differential evolution algorithm, that can calibrate galaxy mocks in a fully automated way for realistic scenarios with a large parameter space. We apply the pipeline to galaxy mocks built on a combination of halo occupation distribution and sub-halo abundance matching techniques. We show that our pipeline can properly calibrate the galaxy mocks against observations for both $\Lambda$CDM and modified gravity halo catalogues. This type of calibration pipeline provides a new tool for automating the calibration of future massive galaxy mocks.

Figures

Figures reproduced from arXiv: 2507.23451 by the authors.

Figure 1
Figure 1. Histograms of the χ 2 value as a function of the number of evalua￾tions. The absolute minimum of the χ 2 is represented by the red triangle. The stochastic behaviour of the problem can be observed, as well as the convergence of χ 2 values as a function of the number of evaluations. Top panel: ΛCDM catalogue. Bottom panel: Modified gravity f(R) cat￾alogue. The colour map corresponds to a linear mapping between 0 and … view at source ↗
Figure 2
Figure 2. Comparison of the projected two-point correlation function for different luminosity thresholds. The measurements on real observations are represented by solid lines, while the corresponding measurements on the calibrated ΛCDM mocks are represented by dots. The agreement between observations and mocks is good, in particular on scales involv￾ing the 1-halo term (smaller than 1 h −1 Mpc). 2021). We only need to ensure … view at source ↗
Figure 4
Figure 4. Same as in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figures from the paper (4 more)
Figure 3
Figure 3. Figure 3: Same as in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png]
Figure 6
Figure 6. Figure 6: Comparison between the calibrated parameters for the general relativity (GR) and modified gravity (MG) mocks. In this panel we present the impact of different values for α, ai , bj , and sk (see [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 5
Figure 5. Figure 5: Same as in [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Comparison between the calibrated parameters for the general relativity (GR) and modified gravity (MG) mocks. In this panel we present the impact of different values for aAM, bAM, and cAM (see Ta￾ble 1) determining the relation between the characteristic luminosity and…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Euclid. Populating a dark universe with galaxies using SciPIC

    astro-ph.CO 2026-04 unverdicted novelty 5.0 of 10

    SciPICal calibrates galaxy properties in halo-based mocks for Euclid, delivering improved clustering predictions and a deep catalogue validated to 15 percent against observations.

Reference graph

Works this paper leans on

30 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Abbott, T. M. C., Aguena, M., Alarcon, A., et al. 2022, PRD, 105, 023520

  2. [2]

    G., Aguilar, J., Ahlen, S., et al

    Adame, A. G., Aguilar, J., Ahlen, S., et al. 2025, JCAP, 02, 021

  3. [3]

    2021, MNRAS, 504, 4667

    Alam, S., de Mattia, A., Tamone, A., et al. 2021, MNRAS, 504, 4667

  4. [4]

    2019, MNRAS, 483, 790

    Arnold, C., Fosalba, P., Springel, V ., Puchwein, E., & Blot, L. 2019, MNRAS, 483, 790

  5. [5]

    Benson, A. J. 2012, New A, 17, 175

  6. [6]

    R., Lupton, R

    Blanton, M. R., Lupton, R. H., Schlegel, D. J., et al. 2005, ApJ, 631, 208

  7. [7]

    R., Schlegel, D

    Blanton, M. R., Schlegel, D. J., Strauss, M. A., et al. 2005, AJ, 129, 2562

  8. [8]

    Buchdahl, H. A. 1970, MNRAS, 150, 1

Show all 30 references
  1. [9]

    J., Gaztañaga, E., Crocce, M., & Fosalba, P

    Carretero, J., Castander, F. J., Gaztañaga, E., Crocce, M., & Fosalba, P. 2015, MNRAS, 447, 646

  2. [10]

    2017, in Proceedings of the European Physical Society Conference on High Energy Physics

    Carretero, J., Tallada, P., Casals, J., et al. 2017, in Proceedings of the European Physical Society Conference on High Energy Physics. 5-12 July, 488

  3. [11]

    J., Gaztañaga, E., Fosalba, P., & Carretero, J

    Crocce, M., Castander, F. J., Gaztañaga, E., Fosalba, P., & Carretero, J. 2015, MNRAS, 453, 1513

  4. [12]

    H., Becker, M

    DeRose, J., Wechsler, R. H., Becker, M. R., et al. 2022, PRD, 105, 123520 Euclid Collaboration: Castander, F. J., Fosalba, P., Stadel, J., et al. 2025, A&A, 697, A5 Euclid Collaboration: Knabenhans, M., Stadel, J., Potter, D., et al. 2021, MN- RAS, 505, 2840

  5. [13]

    2021, A&A, 656, A106

    Ferrero, I., Crocce, M., Tutusaus, I., et al. 2021, A&A, 656, A106

  6. [14]

    2021, A&A, 646, A140

    Heymans, C., Tröster, T., Asgari, M., et al. 2021, A&A, 646, A140

  7. [15]

    & Sawicki, I

    Hu, W. & Sawicki, I. 2007, PRD, 76, 064004

  8. [16]

    & Roos, M

    James, F. & Roos, M. 1975, Comput. Phys. Commun., 10, 343

  9. [17]

    2001, SciPy: Open source scientific tools for Python

    Jones, E., Oliphant, T., Peterson, P., et al. 2001, SciPy: Open source scientific tools for Python

  10. [18]

    F., Frenk, C

    Navarro, J. F., Frenk, C. S., & White, S. D. M. 1997, ApJ, 490, 493

  11. [19]

    2013, MNRAS, 436, 348

    Puchwein, E., Baldi, M., & Springel, V . 2013, MNRAS, 436, 348

  12. [20]

    2020, MNRAS, 499, 269

    Smith, A., Burtin, E., Hou, J., et al. 2020, MNRAS, 499, 269

  13. [21]

    2005, MNRAS, 364, 1105

    Springel, V . 2005, MNRAS, 364, 1105

  14. [22]

    2010, ARA&A, 48, 391

    Springel, V . 2010, ARA&A, 48, 391

  15. [23]

    2008, MNRAS, 391, 1685

    Springel, V ., Wang, J., V ogelsberger, M., et al. 2008, MNRAS, 391, 1685

  16. [24]

    & Price, K

    Storn, R. & Price, K. 1997, J. Glob. Optim., 11, 341

  17. [25]

    2020, A&C, 32, 100391

    Tallada, P., Carretero, J., Casals, J., et al. 2020, A&C, 32, 100391

  18. [26]

    2025, arXiv e-prints, arXiv:2504.04961

    Viglione, C., Fosalba, P., Tutusaus, I., et al. 2025, arXiv e-prints, arXiv:2504.04961

  19. [27]

    F., Berlind, A

    Watson, D. F., Berlind, A. A., McBride, C. K., Hogg, D. W., & Jiang, T. 2012, ApJ, 749, 83

  20. [28]

    Wechsler, R. H. & Tinker, J. L. 2018, ARA&A, 56, 435

  21. [29]

    M., & Bowen, D

    Wormington, M., Panaccione, C., Matney, K. M., & Bowen, D. K. 1999, Phil. Trans. R. Soc. A, 357, 2827

  22. [30]

    H., et al

    Zehavi, I., Zheng, Z., Weinberg, D. H., et al. 2011, ApJ, 736, 59 Article number, page 10 of 10

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.