Towards a Precise Measurement of Matter Clustering: Lyman-alpha Forest Data at Redshifts 2-4

We measure the filling factor, correlation function, and power spectrum of transmitted flux in a large sample of Lya forest spectra, comprised of 30 Keck HIRES spectra and 23 Keck LRIS spectra. We infer the linear matter power spectrum P(k) from the flux power spectrum P_F(k), using an improved version of the method of Croft et al. (1998) that accounts for the influence of z-space distortions, non- linearity, and thermal broadening on P_F(k). The evolution of the shape and amplitude of P(k) over the range z= 2-4 is consistent with gravitational instability, implying that non-gravitational fluctuations do not make a large contribution. Our fiducial measurement of P(k) comes from data with <z> = 2.72. It has amplitude Delta^2(k_p)=0.74^0.20_-0.16 at wavenumber k_p=0.03 (km/s)^-1 and is well described by a power-law of index -2.43 +/- 0.06 or by a CDM-like power spectrum with shape parameter Gamma'=1.3^+0.7_-0.5*10^-3 (km/s) at z=2.72. For Omega_m=0.4, Omega_Lam=0.6, the best-fit Gamma =0.16 (h^-1mpc)^-1, in good agreement with the 2dF Galaxy Redshift Survey, and the best-fit sigma_8=0.82 (Gamma/0.15)^-0.44. Matching the observed cluster mass function and our Delta^2(k_p) in spatially flat models requires Omega_m=0.38^+0.10_-0.08 + 2.2 (Gamma-0.15). Matching Delta^2(k_p) in COBE-normalized, flat CDM models with no tensor fluctuations requires Omega_m = (0.29 +/-0.04) n^-2.89 h_65^-1.9. The Lya forest complements other probes of P(k) by constraining a regime of redshift and lengthscale not accessible by other means, and the consistency of these inferred parameters with independent estimates provides further support for inflation, cold dark matter, and vacuum energy (abridged).