REVIEW 3 major objections 3 minor 13 references
Augur Fisher forecasts reproduce full posterior contours for LSST 3x2pt analyses, the paper shows, once its numerical settings are stabilized.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
Augur is a validated DESC Fisher-forecasting pipeline whose Y1/Y10 3x2pt constraints match nested sampling and external forecast codes when derivative step sizes and redshift sampling are chosen in a stable regime.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection Worth a serious referee: the validation is real, but the 500-knot claim conflicts with the 1000-knot caption in Fig. 6. the 3 major comments →
Fisher Forecasting for the DESC with $\texttt{Augur}$
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The paper establishes a reliable operating point for Augur: normalized derivative step size of 5% of the uniform prior range for every parameter, evaluated with the 5-point stencil or numdifftools, and n(z) represented by roughly 500 spline knots. At this point the Fisher matrix approximates the local likelihood well enough that its full correlation matrix sits within 0.05 of the nested-sampling correlation matrix for both Y1 and Y10 setups, its w0-wa degeneracy direction matches sampling, and its Dark Energy Figure of Merit agrees with the DESC SRD pipeline to 10% (Y1) and 2% (Y10). The paper also characterizes unstable regimes: at smaller step sizes, oscillatory features in non-linear matt
What carries the argument
The engine is the Fisher information matrix evaluated from numerical derivatives of the harmonic-space angular power spectrum data vector, F_alpha_beta = (df/dtheta_alpha)^T C^{-1} (df/dtheta_beta), evaluated at the fiducial cosmology under a Gaussian likelihood with parameter-independent covariance. The practical workhorse is the 5-point stencil derivative; the step size is normalized to 5% of each parameter's uniform prior extent, and the redshift distribution is supplied as a spline over roughly 500 knots. These two choices control the trade-off between truncation error and numerical noise that dominates Fisher computations for cosmological two-point functions, and the paper's stability t
Load-bearing premise
The whole validation rests on the log-posterior being locally quadratic at the fiducial cosmology and on the data covariance being independent of the cosmological parameters; if either fails, the Fisher contours will misstate the posterior even in the numerically stable regime.
What would settle it
Re-run the Augur-versus-PolyChord comparison at a shifted fiducial cosmology, say w0 = -1.3 with the other parameters rescaled consistently, and check whether the correlation-matrix difference stays below 0.1 and the DEFOM within 20%. If those criteria fail, the claimed stable configuration is specific to the fiducial used here rather than a general property of the pipeline.
If this is right
- Users can replace expensive MCMC or nested-sampling scans with Augur Fisher forecasts for LSST Y1/Y10 3x2pt setups, provided they adopt the documented stable step sizes and n(z) sampling.
- Forecasts, likelihoods, and covariances now share one ecosystem, so Fisher studies can be compared directly with the actual analysis pipeline rather than a separate forecasting code.
- Fisher bias estimates can be trusted for small model mis-specifications, up to about 1 sigma in w0 and 0.5 sigma in wa, providing a fast first-pass diagnostic for systematics.
- Forecasts produced with sub-percent derivative step sizes or sparse n(z) sampling should be treated as unreliable; the paper's thresholds give a concrete way to check stability before interpreting results.
- Bin-averaging bandpowers stabilizes derivative behavior and changes the DEFOM through scale-cut effects, so binning choices must be reported and tested alongside any Fisher forecast.
Where Pith is reading between the lines
- The validation is fiducial-specific: nothing in the paper demonstrates that the 5%-step / 500-knot recipe transfers to other fiducial cosmologies or to likelihoods with parameter-dependent covariance, so users of Augur should re-run the stability tests for each new model.
- A natural next test would be repeating the nested-sampling comparison at a shifted fiducial, such as w0 = -1.3, to map how far the linearity region extends; the paper's own procedures provide the tool for that test.
- The breakdown of Fisher bias beyond roughly 1 sigma suggests a hybrid workflow: use Augur for fast screening, then invoke nested sampling or MCMC only in parameter regions where the bias checks fail the 10% criterion.
- The sensitivity of Halofit's oscillatory derivatives to step size points toward automatic differentiation or analytic derivative models as a way to push stable Fisher forecasts to smaller scales than finite differencing allows.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents Augur, a Fisher forecasting package for the LSST DESC software ecosystem, built on CCL, TJPCov, and firecrown. The authors perform a validation exercise for 3x2pt harmonic-space analyses with LSST Y1/Y10-like setups, testing the stability of Fisher matrices against derivative step size, n(z) knot sampling, matter power spectrum prescription, and bandpower binning. They compare Augur's Fisher contours, correlation matrices, and figures of merit against a CosmoSIS+firecrown Fisher implementation, the DESC SRD pipeline, and direct PolyChord nested sampling, reporting good agreement within the thresholds defined in Section 4. They also test the Fisher bias formalism with one-dimensional parameter shifts. The central claim is that, when operated in a numerically stable configuration, Augur reliably reproduces both external Fisher pipelines and full posterior sampling.
Significance. If the validation holds, Augur provides a useful, publicly available forecasting component for DESC cosmological analyses. The paper's strongest contributions are its external check against nested sampling (Figures 4 and 5), the systematic mapping of numerical hyperparameter sensitivity (Figures 1-3 and 6-7), and the explicit documentation of stability criteria for Fisher-matrix users. The work is methodologically valuable for the DESC software stack and for future Fisher analyses of LSST data. The code is publicly available, which aids reproducibility. The comparison with CosmoSIS is limited by the shared firecrown likelihood, but the paper is appropriately cautious in describing it as a validation of the Fisher implementation rather than of the physical model.
major comments (3)
- [Section 5.2 / Figure 6] There is an internal contradiction in the recommended n(z) knot configuration. Section 5.2 states that 'convergence is achieved when the n(z) distribution is sampled with several hundred knots' and adopts 500 knots, while the caption of Figure 6 states that 'stable solutions are generally found when increasing the number of knots in the n(z) distribution to nearly 1000 evaluations, regardless of the analysis year considered.' Since the headline validations in Figures 4, 5, and 8 use 500 knots, this is load-bearing. Please report the DEFOM and Fisher-matrix differences between 500 and 1000+ knots against the Section 4 thresholds; if 500 is not in the converged regime, rerun the comparisons at a converged knot number and update the recommended configuration.
- [Section 5.4 / Figure 8] The claimed Y1 DEFOM agreement with the SRD pipeline is not supported by the numbers in Figure 8. The caption lists DEFOM = 36.95 for the DESC SRD Pipeline and 41.36 for Augur (and CosmoSIS), a relative difference of about 12%, which exceeds the 10% agreement threshold stated in Section 4 and the 'within 10%' claim in Section 5.4 and the Conclusion. Please correct either the numbers or the text, and ensure the reported Y1 agreement is consistent with the stated threshold.
- [Section 4, stability criterion] The stability criterion is written as (F^{Δs}_{αβ} - F^{Δs'}_{αβ})/F^{Δs'}_{αβ} ≤ 0.1 without an absolute value. As written, any negative relative difference satisfies the inequality, so it does not enforce 'does not change by more than 10%' in the sense intended. If the implementation uses an absolute value, this should be stated explicitly. This matters because the stable configuration chosen in Section 5.1 is selected using this criterion.
minor comments (3)
- [Figure 4] The legend text in the figure says 'CosmoSIS+ firecrown Y1 Nested Sampling', but the caption and text attribute the nested sampling to PolyChord. Since the nested sampling run uses the firecrown likelihood but is not a CosmoSIS Fisher run, the legend label should be changed to 'PolyChord nested sampling' for clarity.
- [Figure 6] The caption for Figure 6b contains a stray 'the' at the end: 'for the number of knots evaluated in the n(z) distribution. the'.
- [Section 2.1] The paper correctly notes that Eq. (4) assumes a parameter-independent covariance. This is an important limitation, particularly for future analyses using model covariances; it would be helpful to state explicitly in the conclusions that this assumption was not varied in the validation.
Circularity Check
No significant circularity: Augur's Fisher forecasts are validated against independent nested sampling and external pipelines; the n(z)-knot caption discrepancy is a stability concern, not circularity.
full rationale
Augur's validation chain is not circular. The Fisher matrix (Sec. 2.1, Eq. 4) is computed by finite-differencing the firecrown/CCL model prediction at the fiducial point; the target comparisons are (i) PolyChord nested sampling of the same likelihood (Figs. 4-5), (ii) CosmoSIS+firecrown using a different derivative implementation, and (iii) the DESC SRD pipeline built on CosmoLike, an independent code. None of these targets is used to define the Fisher matrix or to select the stability criteria: the step size Δs=0.05 and the 500-knot n(z) sampling are chosen by the paper's own local-stability tests (relative Fisher-matrix differences ≤10% between adjacent step sizes; DEFOM plateau in Sec. 5.2), not by fitting to the nested-sampling or SRD contours. The paper explicitly restricts the claim: 'This comparison therefore serves not as a validation of the Fisher methodology itself, but as confirmation that Augur's implementation reproduces full-likelihood results when evaluated within its expected domain of stability.' The Fisher-bias recovery test (Sec. 5.5) is an internal code self-check and is presented as such, not as an independent prediction. Self-citations to DESC tools (CCL, firecrown, TJPCov, SRD) are code dependencies; their outputs are externally checked by the CosmoLike-based SRD pipeline. The only substantive concern is an internal inconsistency: Sec. 5.2 concludes convergence at 'several hundred knots' and adopts 500, while the Fig. 6 caption states 'Stable solutions are generally found when increasing the number of knots in the n(z) distribution to nearly 1000 evaluations.' This is a hyperparameter-convergence issue that could affect numerical reliability of the headline comparisons, but it is not a circular derivation of the results from their inputs.
Axiom & Free-Parameter Ledger
free parameters (2)
- Normalized derivative step size Delta s =
0.05 (5% of the extent of the uniform prior range)
- n(z) sampling density =
500 knots
axioms (7)
- domain assumption The log-posterior is locally quadratic around the fiducial model, so the Fisher matrix encodes the covariance (Cramer-Rao bound saturated).
- domain assumption The data likelihood is Gaussian with a parameter-independent covariance matrix.
- domain assumption The Limber approximation is valid for the chosen ell bins (ell >= 20).
- domain assumption Linear galaxy bias per tomographic bin and the NLA intrinsic alignment model are sufficient at the chosen scale cuts.
- domain assumption The chosen non-linear matter power spectrum prescriptions (EH+Halofit fiducial, plus CAMB+HMCode2020 and CAMB+Halofit) are accurate.
- domain assumption The DESC SRD covariance matrix is an adequate fixed covariance for Y1/Y10 3x2pt forecasts.
- ad hoc to paper The stability thresholds (relative Fisher difference <= 10%, correlation matrix difference <= 0.1, FOM difference <= 20%) are adequate measures of good agreement.
Cite this review
Pith. "Pith review of Fisher Forecasting for the DESC with $\texttt{Augur}$." pith.science (2026). https://pith.science/paper/BELET4L5
@misc{pith2026260803876,
author = {Pith},
title = {Pith review of: Fisher Forecasting for the DESC with $\textttAugur$},
year = {2026},
howpublished = {\url{https://pith.science/paper/BELET4L5}},
note = {Machine review of arXiv:2608.03876}
}
abstract
The Vera C. Rubin Observatory Legacy Survey of Space and Time (LSST) has begun its ten-year survey of the entire visible southern hemisphere. To ensure robust cosmological measurements, computationally inexpensive investigations of modeling choices must be made to gauge the performance of proposed cosmological analyses. In this paper, we introduce the $\texttt{Augur}$ tool of the Dark Energy Science Collaboration (DESC), which provides Fisher forecasts for cosmological inference for the LSST using software frameworks designed for DESC science. We test the pipeline by comparing it to forecasts produced by external code and direct sampling of the posterior via nested sampling methods, finding good agreement between all methods. We additionally investigate a range of modeling and hyperparameter choices for a 3$\times$2pt investigation in harmonic space, providing users with diagnostics to obtain reliable forecasts. $\texttt{Augur}$ will be continually updated to be compatible with the other tools in the DESC software ecosystem as additional probes and functionality become available.
Figures
Reference graph
Works this paper leans on
-
[1]
A. G. Riess, A. V. Filippenko, P. Challis, A. Clocchiatti, A. Diercks, P. M. Garnavich, R. L. Gilliland, C. J. Hogan, S. Jha, R. P. Kirshner, B. Leibundgut, M. M. Phillips, D. Reiss, B. P. Schmidt, R. A. Schommer, R. C. Smith, J. Spyromilio, C. Stubbs, N. B. Suntzeff, and J. Tonry, Astrophysical Journal 116, 1009 (1998), arXiv:astro-ph/9805201 [astro-ph]....
Pith/arXiv arXiv 1998
-
[2]
Report of the dark energy task force,
Mather, S. Staggs, and N. B. Suntzeff, “Report of the dark energy task force,” (2006), arXiv:astro-ph/0609591 [astro-ph]. B. Abareshiet al., The Astronomical Journal164, 207 (2022). Y. Mellieret al., Astronomy &; Astrophysics697, A1 (2025). H. Ausselet al., “Euclid quick data release (q1) – data release overview,” (2025), arXiv:2503.15302 [astro-ph.GA]. N...
Pith/arXiv arXiv 2006
-
[3]
Sanchez, A. Slosar, M. Sullivan, and M. A. Troxel, arXiv e-prints , arXiv:1809.01669 (2018), arXiv:1809.01669 [astro-ph.CO]. N. E. Chisari, D. Alonso, E. Krause, C. D. Leonard, P. Bull, J. Neveu, A. Villarreal, S. Singh, T. McClintock, J. Ellison, Z. Du, J. Zuntz, A. Mead, S. Joudaki, C. S. Lorenz, T. Tr¨ oster, J. Sanchez, F. Lanusse, M. Ishak, R. Hlozek...
Pith/arXiv arXiv 2018
-
[4]
Campagne, H. Almoubayyed, T. Eifler, M. Kirby, D. Kirkby, S. Plaszczynski, A. Slosar, M. Vrastil, and E. L. Wagoner, The Astrophysical Journal Supplement Series242, 2 (2019a). N. Bhandari, C. D. Leonard, M. M. Rau, and R. Mandelbaum, arXiv e-prints , arXiv:2101.00298 (2021), arXiv:2101.00298 [astro-ph.CO]. D. Huterer and M. Takada, Astroparticle Physics23...
work page internal anchor Pith review Pith/arXiv arXiv 2021
-
[5]
Kisner, R. Kneissl, J. Knoche, L. Knox, M. Kunz, H. Kurki-Suonio, G. Lagache, A. L¨ ahteenm¨ aki, J.-M. Lamarre, A. Lasenby, M. Lattanzi, C. R. Lawrence, J. P. Leahy, R. Leonardi, J. Lesgourgues, F. Levrier, A. Lewis, M. Liguori, P. B. Lilje, M. Linden-Vørnle, M. L´ opez-Caniego, P. M. Lubin, J. F. Mac ´ ıas-P´ erez, G. Maggio, D. Maino, N. Mandolesi, A. ...
Pith/arXiv arXiv 2016
-
[6]
Price, P. J. Tait, M. Tanaka, and S.-Y. Wang, Physical Review D108, 123519 (2023), arXiv:2304.00701 [astro-ph.CO]. A. H. Wright, B. St¨ olzner, M. Asgari, M. Bilicki, B. Giblin, C. Heymans, H. Hildebrandt, H. Hoekstra, B. Joachimi, K. Kuijken, S.-S. Li, R. Reischke, M. von
Pith/arXiv arXiv 2023
-
[7]
Napolitano, L. Porth, M. Radovich, P. Schneider, T. Tr¨ oster, E. Valentijn, A. Wittje, Z. Yan, and Y.-H. Zhang, Astronomy and Astrophysics703, A158 (2025), arXiv:2503.19441 [astro-ph.CO]. 19 DES Collaboration, T. M. C. Abbott, M. Adamow, M. Aguena, A. Alarcon, S. S. Allam, O. Alves, A. Amon, D. Anbajagane, F. Andrade-Oliveira, S. Avila, D. Bacon, E. J. B...
Pith/arXiv arXiv 2025
-
[8]
Wu, M. Yamamoto, B. Yanny, B. Yin, G. Zacharegkas, Y. Zhang, and J. Zuntz, arXiv e-prints , arXiv:2601.14559 (2026), arXiv:2601.14559 [astro-ph.CO]. A. Nicola, B. Hadzhiyska, N. Findlay, C. Garc ´ ıa-Garc ´ ıa, D. Alonso, A. Slosar, Z. Guo, N. Kokron, R. Angulo, A. Aviles, J. Blazek, J. Dunkley, B. Jain, M. Pellejero, J. Sullivan, C. W
arXiv 2026
-
[9]
Collaboration, Journal of Cosmology and Astroparticle Physics 2024, 015 (2024), arXiv:2307.03226 [astro-ph.CO]. H. Awan, E. Gawiser, J. Sanchez, I. Sevilla-Noarbe, and t. L. D. E. S. collaboration, Journal of Cosmology and Astroparticle Physics2025, 064 (2025). N. E. Chisari, A. J. Mead, S. Joudaki, P. G. Ferreira, A. Schneider, J. Mohr, T. Tr¨ oster, D. ...
Pith/arXiv arXiv 2024
-
[10]
CAMB: Code for Anisotropies in the Microwave Background,
Daalen, The Open Journal of Astrophysics2(2019b), 10.21105/astro.1905.06082. D. N. Limber, Astrophysical Journal117, 134 (1953). P. Lemos, A. Challinor, and G. Efstathiou, Journal of Cosmology and Astroparticle Physics2017, 014–014 (2017). I. Smail, R. S. Ellis, and M. J. Fitchett, Monthly Notices of the Royal Astronomical Society270, 245 (1994), arXiv:as...
Pith/arXiv arXiv 1905
-
[11]
Cobra: Optimal factorization of cosmological observables,
Sciences, St. Petersburg, 1853). T. Bakx, N. E. Chisari, and Z. Vlah, “Cobra: Optimal factorization of cosmological observables,” (2025a), arXiv:2407.04660 [astro-ph.CO]. T. Bakx, H. Rubira, N. E. Chisari, and Z. Vlah, (2025b), arXiv:2508.00611 [astro-ph.CO]. G. D’Amico, A. Refregier, L. Senatore, and P. Zhang, “The cosmological analysis of des 3×2pt data...
Pith/arXiv arXiv 2025
-
[12]
Heavens, Monthly Notices of the Royal Astronomical Society 454, 1958 (2015), https://academic.oup.com/mnras/article- pdf/454/2/1958/9381114/stv2036.pdf. D. J. Bartlett, B. D. Wandelt, M. Zennaro, P. G. Ferreira, and H. Desmond, Astronomy and Astrophysics686, A150 (2024), arXiv:2402.17492 [astro-ph.CO]. G. Aric` o, R. E. Angulo, S. Contreras, L. Ondaro-Mal...
Pith/arXiv arXiv 1958
-
[13]
Archibald, A. H. Ribeiro, F. Pedregosa, P. van Mulbregt, and SciPy 1.0 Contributors, Nature Methods17, 261 (2020). J. D. Hunter, Computing in Science & Engineering9, 90 (2007). A. Lewis, Journal of Cosmology and Astroparticle Physics2025, 025 (2025) 20 This paper was built using the Open Journal of As- trophysics LATEX template. The OJA is a journal which...
work page 2020
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.