Vacuum squeezing via polarization self-rotation and excess noise in hot Rb vapors

We present experimental and theoretical analysis of quantum fluctuation in a vacuum field in the presence of orthogonal linearly polarized pump field propagating through a Rb vapor cell. Previously reported theoretical and experimental studies provided somewhat contradictory conclusions regarding the possibility to observe the "squeezed vacuum" -- the reduction of vacuum fluctuations below standard quantum limit -- in this system. Here, using the D1 transitions of Rb in a cell without buffer as as an example, we demonstrate that vacuum squeezing is corrupted by incoherent processes (such as spontaneous emission, elastic scattering, etc.), and its observation is only possible in a specific small region of the experimental parameter space. Numerical simulations, in good agreement with the experiment, demonstrate that the two excited state hyperfine levels play a crucial role in the squeezing and excess noise production. The significant influence of far-detuned atoms on the field fluctuations at low noise frequencies imposes the explicit consideration of the full velocity distribution of the atomic vapor.