Why your model parameter confidences might be too optimistic -- unbiased estimation of the inverse covariance matrix
read the original abstract
AIMS. The maximum-likelihood method is the standard approach to obtain model fits to observational data and the corresponding confidence regions. We investigate possible sources of bias in the log-likelihood function and its subsequent analysis, focusing on estimators of the inverse covariance matrix. Furthermore, we study under which circumstances the estimated covariance matrix is invertible. METHODS. We perform Monte-Carlo simulations to investigate the behaviour of estimators for the inverse covariance matrix, depending on the number of independent data sets and the number of variables of the data vectors. RESULTS. We find that the inverse of the maximum-likelihood estimator of the covariance is biased, the amount of bias depending on the ratio of the number of bins (data vector variables), P, to the number of data sets, N. This bias inevitably leads to an -- in extreme cases catastrophic -- underestimation of the size of confidence regions. We report on a method to remove this bias for the idealised case of Gaussian noise and statistically independent data vectors. Moreover, we demonstrate that marginalisation over parameters introduces a bias into the marginalised log-likelihood function. Measures of the sizes of confidence regions suffer from the same problem. Furthermore, we give an analytic proof for the fact that the estimated covariance matrix is singular if P>N.
This paper has not been read by Pith yet.
Forward citations
Cited by 16 Pith papers
-
Fewer simulations, sharper covariances: Reducing mock covariance noise with Zeldovich approximation control variates
Control variates with Zeldovich mocks reduce covariance matrix variance by up to an order of magnitude on large scales in DESI-like mocks.
-
Testing parity with composite-field spectra of BOSS and DESI luminous red galaxies
No evidence for cosmological parity violation is found in the first kurto-spectrum analysis of BOSS DR12 and DESI DR1 luminous red galaxies.
-
Nearest Neighbour-Based Statistics for 21cm-Galaxy Cross-Correlations in the Epoch of Reionization
kNN CDF statistics detect 21cm-galaxy cross-correlations more effectively than two-point methods and distinguish reionization models at fixed ionized fraction even with noise and foregrounds.
-
The 3D clustering of Lyman Alpha Emitters measured with DESI
DESI LAE clustering measurements give a linear bias of 2.31-2.62 with constraints on radiative transfer effects and halo occupation from correlation functions and power spectra.
-
Constraints on Halo Gas Profiles from Joint kSZ and Galaxy Clustering Analysis of ACT DR6 and CMASS
Joint kSZ-clustering analysis of ACT DR6 and CMASS yields a halo optical depth profile more extended than Websky simulations, suggesting stronger baryonic feedback.
-
Probing the limits of cosmological information from the Lyman-$\alpha$ forest 2-point correlation functions
Using idealized synthetic data, knowing the true continuum in Lyα forest auto- and cross-correlations reduces uncertainties on the AP parameter and Ω_m by ~10%, with extension to 240 h^{-1}Mpc scales adding up to ~15%...
-
DESI 2024 V: Full-Shape Galaxy Clustering from Galaxies and Quasars
DESI DR1 full-shape galaxy clustering constrains Omega_m = 0.296 ± 0.010, H0 = 68.63 ± 0.79 km/s/Mpc, and sigma_8 = 0.841 ± 0.034, consistent with LambdaCDM and Planck.
-
Modeling and measuring the anisotropic halo 3-point correlation function: a coordinated study
Novel implementation of anisotropic 3PCF model and estimator tested on 298 halo catalogs shows degeneracy breaking between f and b1 in 3PCF-only analysis but limited added value in joint 2PCF+3PCF due to tree-level mo...
-
Cosmology-dependent covariance in galaxy cluster number counts: consequences for parameter inference
Fixing the covariance at an incorrect cosmology in cluster count analyses leaves Ω_c, σ_8, and w estimates unbiased but distorts their uncertainties, driven by S_8 amplitude effects; a single update at the recovered b...
-
Revisiting the 'Lensing is Low' Problem with UNIONS
New UNIONS galaxy-galaxy lensing data around CMASS galaxies indicates no significant lensing is low problem, with joint HOD fits to GGL and GC favoring a slightly lower matter power spectrum amplitude than Planck.
-
(The) Wiggles going non-linear
N-body simulations calibrate a one-parameter damping model that predicts the non-linear matter power spectrum from wiggly primordial spectra to sub-percent accuracy when wiggle frequency is high enough.
-
FolpsD: combining EFT and phenomenological approaches for joint power spectrum and bispectrum analyses
FolpsD combines EFT power spectrum and tree-level bispectrum with damping to enable joint analyses that improve cosmological constraints from DESI-like galaxy mocks by up to 30% on As and omega_cdm while extending the...
-
Counting voids and filaments: Betti Curves as a Powerful Probe for Cosmology
Betti curves from persistent homology of large-scale structure provide complementary cosmological constraints on ns, sigma8, and Om, with tighter bounds when analyzed jointly with the power spectrum.
-
Constraints on primordial non-Gaussianity from Quaia
Reports f_NL = -20.5^{+19.0}_{-18.1} (68% CL) from combined Quaia quasar auto-correlation and CMB lensing cross-correlation assuming p_phi=1, or -28.7^{+26.1}_{-24.6} for p_phi=1.6.
-
Tracing Large-scale Structure with the MeerKLASS On-the-Fly Survey: Angular Clustering of Radio Sources at 816 MHz
First measurement of angular clustering w(theta) for radio sources at 816 MHz over 800 deg2 yields positive signal and effective bias 1.53-2.0 depending on N(z) prior.
-
Cosmic Shear constraints from HSC Year 3 with clustering calibration of the tomographic redshift distributions from DESI
Reanalysis of HSC Y3 cosmic shear with DESI clustering redshift calibration yields S8 = 0.805 ± 0.018, a 1.8× error reduction and upward shift toward Planck cosmology.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.