Quantitative investigation of the short-range magnetic correlations in candidate quantum spin liquid NaYbO$_2$

We present a neutron diffraction study of NaYbO$_2$, a candidate quantum spin liquid compound hosting a geometrically frustrated triangular lattice of magnetic Yb$^{3+}$ ions. We observe diffuse magnetic scattering that persists to at least 20 K, demonstrating the presence of short-range magnetic correlations in this system up to a relatively high energy scale. Using reverse Monte Carlo and magnetic pair distribution function analysis, we confirm the predominant antiferromagnetic nature of these correlations and show that the diffuse scattering data can be well described by noninteracting layers of XY spins on the triangular lattice. We rule out Ising spins and short-range-ordered stripe or 120$^{\circ}$ phases as candidate ground states of NaYbO$_2$. These results are consistent with a possible QSL ground state in NaYbO$_2$ and showcase the benefit of combined reciprocal- and real-space analysis of materials with short-range magnetic correlations.