Observation of an anomalous density-dependent energy gap of the ν=5/2 fractional quantum Hall state in the low density regime

We have studied the $\nu=5/2$ fractional quantum Hall state in a density-tunable sample at extremely low electron densities. For the densities accessed in our experiment, the Landau level mixing parameter $\kappa$ spans the $2.52<\kappa<2.82$ range. In the vicinity of $5.8 \times 10^{10}$~cm$^{-2}$ or $\kappa = 2.6$ an anomalously large change in the density dependence of the energy gap is observed. Possible origins of such an anomaly are discussed, including a topological phase transition in the $\nu=5/2$ fractional quantum Hall state.