Euclid preparation. LXXXIX. Accurate and precise data-driven angular power spectrum covariances
Pith reviewed 2026-05-19 10:01 UTC · model grok-4.3
The pith
DICES combines jackknife resampling with shrinkage to the Gaussian prediction to produce accurate non-singular covariances for angular power spectra.
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
DICES produces accurate, non-singular covariance estimates for Euclid's anticipated large data vector by estimating a noisy covariance through conventional delete-1 jackknife resampling, followed by linear shrinkage of the empirical correlation matrix towards the Gaussian prediction, and then applying a delete-2 jackknife bias correction to the diagonal components. This approach removes the bias common in jackknife error estimates and improves the relative error by 33% for the covariance and 48% for the correlation structure in comparison to jackknife estimates, enabling highly accurate regression and inference without assumptions about cosmology or galaxy bias.
What carries the argument
DICES (Debiased Internal Covariance Estimation with Shrinkage), which shrinks the empirical correlation matrix to the Gaussian prediction and corrects bias using delete-2 jackknife on equal-area segments from binary space partition.
If this is right
- These covariance estimates can be used for highly accurate regression and inference on large data vectors.
- The method is critical for validation against observational systematic effects in Euclid data.
- Non-singular covariances allow reliable computation of likelihoods for parameter estimation.
- Internal derivation requires no external assumptions about cosmology or galaxy populations.
Where Pith is reading between the lines
- The equal-area jackknife segmentation could improve covariance estimates for other surveys using spherical data.
- Shrinkage towards Gaussian might be adapted for other types of power spectrum measurements beyond Euclid.
- Further bias corrections or different shrinkage targets could be tested to optimize for specific data vectors.
Load-bearing premise
The Gaussian prediction used for shrinkage is sufficiently close to the true covariance structure that the linear combination remains unbiased for the Euclid-like data vectors.
What would settle it
Direct comparison of the DICES-estimated covariance matrix against the sample covariance from a large number of independent simulations would reveal if the claimed error reductions hold without introducing new biases.
Figures
read the original abstract
We develop techniques for generating accurate and precise internal covariances for measurements of clustering and weak-lensing angular power spectra. These methods have been designed to produce non-singular and unbiased covariances for Euclid's large anticipated data vector and will be critical for validation against observational systematic effects. We constructed jackknife segments that are equal in area to a high precision by adapting the binary space partition algorithm to work on arbitrarily shaped regions on the unit sphere. Jackknife estimates of the covariances are internally derived and require no assumptions about cosmology or galaxy population and bias. Our covariance estimation, called DICES (Debiased Internal Covariance Estimation with Shrinkage), first estimated a noisy covariance through conventional delete-1 jackknife resampling. This was followed by linear shrinkage of the empirical correlation matrix towards the Gaussian prediction, rather than linear shrinkage of the covariance matrix. Shrinkage ensures the covariance is non-singular and therefore invertible, which is critical for the estimation of likelihoods and validation. We then applied a delete-2 jackknife bias correction to the diagonal components of the jackknife covariance that removed the general tendency for jackknife error estimates to be biased high. We validated internally derived covariances, which used the jackknife resampling technique, on synthetic Euclid-like lognormal catalogues. We demonstrate that DICES produces accurate, non-singular covariance estimates, with the relative error improving by 33% for the covariance and 48% for the correlation structure in comparison to jackknife estimates. These estimates can be used for highly accurate regression and inference.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces the DICES method for internal estimation of covariances for angular power spectra of clustering and weak lensing. It starts with a delete-1 jackknife covariance from equal-area segments generated via adapted binary space partitioning on the sphere, applies linear shrinkage specifically to the empirical correlation matrix toward a Gaussian prediction to guarantee non-singularity, and then applies a delete-2 jackknife correction only to the diagonal elements to reduce the known high bias of jackknife variances. Validation on synthetic Euclid-like lognormal catalogs shows claimed relative-error reductions of 33% for the covariance and 48% for the correlation matrix relative to plain jackknife, enabling accurate regression and inference without external cosmology or bias assumptions.
Significance. If the central accuracy and unbiasedness claims hold under realistic non-Gaussian contributions, the method supplies a practical, fully internal route to invertible covariance matrices for the large data vectors expected from Euclid. The reported error reductions and the avoidance of external modeling assumptions would be useful for systematic validation and likelihood analyses.
major comments (2)
- Abstract and § on DICES procedure: the claim that the final matrices are accurate and unbiased rests on the premise that shrinkage of the correlation matrix toward the Gaussian prediction does not introduce net systematic error when the true covariance contains non-Gaussian contributions (as in the lognormal mocks). Because the delete-2 correction is applied only to the diagonal, any off-diagonal pull from an imperfect Gaussian target remains uncompensated; a quantitative residual-bias test against the ensemble-averaged true covariance must be shown explicitly.
- Validation section: the reported 33% and 48% relative-error improvements are measured on the same synthetic ensemble used to construct or tune the Gaussian target. An independent test (e.g., applying the fixed target derived from one set of mocks to a statistically independent set) is needed to confirm that the improvement does not rely on circularity between target and validation data.
minor comments (2)
- The precise definition of the Gaussian prediction (analytic formula, parameters, or mock-derived) and the method for choosing the shrinkage intensity should be stated with an equation or pseudocode.
- Figure captions and text should explicitly state whether the plotted covariances are normalized or absolute, and whether the reported relative errors are averaged over all matrix elements or only off-diagonal blocks.
Simulated Author's Rebuttal
We thank the referee for their careful and constructive review of our manuscript on the DICES covariance estimation method. We address each major comment in detail below, providing clarifications based on the existing analysis and committing to targeted revisions that will strengthen the validation of accuracy and lack of bias.
read point-by-point responses
-
Referee: Abstract and § on DICES procedure: the claim that the final matrices are accurate and unbiased rests on the premise that shrinkage of the correlation matrix toward the Gaussian prediction does not introduce net systematic error when the true covariance contains non-Gaussian contributions (as in the lognormal mocks). Because the delete-2 correction is applied only to the diagonal, any off-diagonal pull from an imperfect Gaussian target remains uncompensated; a quantitative residual-bias test against the ensemble-averaged true covariance must be shown explicitly.
Authors: We agree that an explicit quantification of any residual bias from the shrinkage step is valuable for readers. Our current validation already compares every DICES matrix element directly to the ensemble-averaged covariance computed from the full set of lognormal mocks; this ensemble average serves as the ground truth and fully incorporates non-Gaussian contributions. The reported 33 % and 48 % relative-error reductions are therefore measured against this non-Gaussian truth. Nevertheless, to isolate the effect of the Gaussian target on off-diagonal elements, we will add a new panel (or subsection) in the validation section that shows the mean fractional bias (DICES minus truth, normalized by truth) separately for diagonal and off-diagonal blocks, both before and after the delete-2 correction. This will demonstrate that any net systematic pull remains sub-dominant to the improvement over plain jackknife. revision: yes
-
Referee: Validation section: the reported 33% and 48% relative-error improvements are measured on the same synthetic ensemble used to construct or tune the Gaussian target. An independent test (e.g., applying the fixed target derived from one set of mocks to a statistically independent set) is needed to confirm that the improvement does not rely on circularity between target and validation data.
Authors: The Gaussian prediction used as the shrinkage target is the analytic Gaussian covariance derived from the survey geometry, mean power spectra, and mask; it is computed once from theoretical expectations and is not fitted or constructed from the mock realizations themselves. Consequently there is no direct circularity in the reported metrics. To address the referee’s concern rigorously, however, we will perform an additional cross-validation experiment in the revised manuscript: the mock ensemble will be partitioned into two statistically independent halves; the analytic Gaussian target will be held fixed (or recomputed only from the first half if any mean-spectrum estimation is involved), and the full DICES procedure plus error-reduction statistics will be evaluated on the held-out half. Results of this test will be reported alongside the original figures. revision: yes
Circularity Check
No significant circularity; DICES estimator is internally derived via jackknife resampling
full rationale
The paper's derivation chain consists of standard delete-1 jackknife resampling to form an initial covariance estimate from the data, followed by linear shrinkage applied specifically to the empirical correlation matrix toward a Gaussian target (used only as a regularizer to ensure non-singularity) and a subsequent delete-2 bias correction restricted to the diagonal. These steps are presented as data-driven with no cosmology or bias assumptions required for the core estimator, and the Gaussian target is not fitted from the target data vector or derived from the final result. Validation on independent synthetic lognormal catalogues shows error reductions relative to plain jackknife, but no equations reduce any claimed prediction or result to the inputs by construction, and no load-bearing self-citations or uniqueness theorems are invoked. The procedure remains self-contained against external benchmarks.
Axiom & Free-Parameter Ledger
axioms (2)
- domain assumption Delete-1 jackknife on equal-area segments yields a usable (if noisy) covariance estimate for angular power spectra.
- domain assumption Linear shrinkage of the empirical correlation matrix toward a Gaussian prediction reduces variance without introducing large bias for Euclid-like data vectors.
Lean theorems connected to this paper
-
IndisputableMonolith/Cost/FunctionalEquation.leanwashburn_uniqueness_aczel unclear?
unclearRelation between the paper passage and the cited Recognition theorem.
DICES first estimated a noisy covariance through conventional delete-1 jackknife resampling. This was followed by linear shrinkage of the empirical correlation matrix towards the Gaussian prediction... We then applied a delete-2 jackknife bias correction to the diagonal components
What do these tags mean?
- matches
- The paper's claim is directly supported by a theorem in the formal canon.
- supports
- The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
- extends
- The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
- uses
- The paper appears to rely on the theorem as machinery.
- contradicts
- The paper's claim conflicts with a theorem or certificate in the canon.
- unclear
- Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.
Forward citations
Cited by 4 Pith papers
-
Fourth-order galaxy-galaxy-lensing: Theoretical framework and direct estimation
The authors derive the fourth-order galaxy-galaxy lensing 4PCF and aperture statistics, implement a numerical pipeline and FFT estimator, and detect the connected ⟨N³ M_ap⟩ signal at SNR ~9 in stage IV mock data over ...
-
If at First You Don't Succeed, Trispectrum: I. Estimating the Matter Power Spectrum Covariance with Higher-Order Statistics
Estimators from squeezed bispectrum and collapsed trispectrum recover unbiased small-scale matter power spectrum covariance at the percent level using 25 Quijote simulations.
-
\textit{Euclid} preparation. Baryon acoustic oscillations extraction techniques: comparison and optimisation
End-to-end validation on Euclid-like mocks shows RecSym and RecIso reconstruction yield unbiased BAO measurements, improving figure of merit for Omega_m and H0 rs by factor of ~3 across 0.9<z<1.8.
-
Euclid preparation. Three-dimensional galaxy clustering in configuration space: Three-point correlation function estimation
Euclid collaboration develops and validates direct and spherical-harmonic estimators plus a random-split optimization for measuring the three-point galaxy correlation function at the scale of the full Euclid survey.
Reference graph
Works this paper leans on
-
[1]
Abbott, T. M. C., Aguena, M., Alarcon, A., et al. 2022, Phys. Rev. D, 105, 023520
work page 2022
-
[2]
Alonso, D., Sanchez, J., Slosar, A., & LSST Dark Energy Science Collaboration. 2019, MNRAS, 484, 4127
work page 2019
-
[3]
Chon, G., Challinor, A., Prunet, S., Hivon, E., & Szapudi, I. 2004, MNRAS, 350, 914
work page 2004
- [4]
-
[5]
Jackknife resampling technique on mocks: an alternative method for covariance matrix estimation
Escoffier, S., Cousinou, M. C., Tilquin, A., et al. 2016, arXiv e-prints, arXiv:1606.00233 5 https://github.com/heracles-ec/heracles Euclid Collaboration: Mellier, Y ., Abdurro’uf, Acevedo Barroso, J., et al. 2025, A&A, 697, A1 Euclid Collaboration: Scaramella, R., Amiaux, J., Mellier, Y ., et al. 2022, A&A, 662, A112 Euclid Collaboration: Tessore, N., Jo...
work page internal anchor Pith review Pith/arXiv arXiv 2016
-
[6]
R., Silva Lafaurie, J., & Sapone, D
Favole, G., Granett, B. R., Silva Lafaurie, J., & Sapone, D. 2021, MNRAS, 505, 5833
work page 2021
-
[7]
Friedrich, O., Seitz, S., Eifler, T. F., & Gruen, D. 2016, MNRAS, 456, 2662
work page 2016
-
[8]
Fuchs, H., Kedem, Z. M., & Naylor, B. F. 1980, SIGGRAPH Comput. Graph., 14, 124–133 García-García, C., Alonso, D., & Bellini, E. 2019, JCAP, 11, 043 Górski, K. M., Hivon, E., Banday, A. J., et al. 2005, ApJ, 622, 759
work page 1980
- [9]
-
[10]
Heymans, C., Tröster, T., Asgari, M., et al. 2021, A&A, 646, A140
work page 2021
- [11]
-
[12]
1931, The Annals of Mathematical Statistics, 2, 360
Hotelling, H. 1931, The Annals of Mathematical Statistics, 2, 360
work page 1931
- [13]
-
[14]
Kwan, J., Sánchez, C., Clampitt, J., et al. 2017, MNRAS, 464, 4045
work page 2017
- [15]
- [16]
-
[17]
Looijmans, M. J., Wang, M. S., & Beutler, F. 2024, arXiv e-prints, arXiv:2402.13783
-
[18]
Loureiro, A., Whittaker, L., Spurio Mancini, A., et al. 2022, A&A, 665, A56
work page 2022
-
[19]
Mohammad, F. G. & Percival, W. J. 2022, MNRAS, 514, 1289
work page 2022
-
[20]
Nicola, A., García-García, C., Alonso, D., et al. 2021, JCAP, 03, 067
work page 2021
-
[21]
M., Gaztañaga, E., & Croton, D
Norberg, P., Baugh, C. M., Gaztañaga, E., & Croton, D. J. 2009, MNRAS, 396, 19
work page 2009
-
[22]
J., Friedrich, O., Sellentin, E., & Heavens, A
Percival, W. J., Friedrich, O., Sellentin, E., & Heavens, A. 2022, MNRAS, 510, 3207 Planck Collaboration, Aghanim, N., Akrami, Y ., et al. 2020, A&A, 641, A6
work page 2022
-
[23]
Pope, A. C. & Szapudi, I. 2008, MNRAS, 389, 766
work page 2008
-
[24]
J., Beutler, F., Chuang, C.-H., et al
Ross, A. J., Beutler, F., Chuang, C.-H., et al. 2017, MNRAS, 464, 1168 Schäfer, J. & Strimmer, K. 2005, Statistical applications in genetics and molec- ular biology, 4
work page 2017
- [25]
-
[26]
Shirasaki, M., Takada, M., Miyatake, H., et al. 2017, MNRAS, 470, 3476
work page 2017
- [27]
-
[28]
Szapudi, I., Prunet, S., Pogosyan, D., Szalay, A. S., & Bond, J. R. 2001, ApJ, 548, L115
work page 2001
- [29]
-
[30]
Tessore, N., Loureiro, A., Joachimi, B., von Wietersheim-Kramsta, M., & Jef- frey, N. 2023, The Open Journal of Astrophysics, 6, 11 1 Department of Physics and Astronomy, University College Lon- don, Gower Street, London WC1E 6BT, UK 2 Institute of Cosmology and Gravitation, University of Portsmouth, Portsmouth PO1 3FX, UK 3 Oskar Klein Centre for Cosmopa...
work page 2023
-
[31]
Padova, Via Marzolo 8, 35131 Padova, Italy 91 Institut de Recherche en Astrophysique et Planétologie (IRAP), Université de Toulouse, CNRS, UPS, CNES, 14 Av. Edouard Belin, 31400 Toulouse, France 92 Université St Joseph; Faculty of Sciences, Beirut, Lebanon 93 Departamento de Física, FCFM, Universidad de Chile, Blanco En- calada 2008, Santiago, Chile 94 Un...
work page 2008
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.