Testing gravity with gauge-invariant polarization states of gravitational waves: Theory and pulsar timing sensitivity

The determination of the polarization modes of gravitational waves (GWs) and their dispersion relations is a crucial task for scrutinizing the viability of extended theories of gravity. A tool to investigate the polarization states of GWs is the well-known formalism developed by Eardley, Lee, and Lightman (ELL) [Phys. Rev. D 8, 3308 (1973)] which uses the Newman-Penrose (NP) coefficients to determine the polarization content of GWs in metric theories of gravity. However, if the speed of GWs is smaller than the speed of light, the number of NP coefficients is greater than the number of polarizations. To overcome this inconvenience we use the Bardeen formalism to describe the six possible polarization modes of GWs considering general dispersion relations for the modes. The definition of a new gauge-invariant quantity enables an unambiguous description of the scalar longitudinal polarization mode. We apply the formalism to General Relativity, scalar-tensor theories, $f(R)$-gravity, and a wide class of quadratic gravity. We derive an explicit relation between a physical observable (the derivative of the frequency shift of an electromagnetic signal), and the gauge-invariant variables. Then we find an analytical formula for the pulsar timing rms response to each polarization mode. To estimate the sensitivity of a single pulsar timing we focus on the case of a dispersion relation of a massive particle. The sensitivity curves of the scalar longitudinal and vector polarization modes change significantly depending on the value of the effective mass. The detection (or absence of detection) of the polarization modes using the pulsar timing technique has decisive implications for alternative theories of gravity. Finally, investigating a cutoff frequency in the pulsar timing band can lead to a more stringent bound on the graviton mass than that presented by ground-based interferometers.