Spectroscopy and modeling of $^{171}$Yb Rydberg states for high-fidelity two-qubit gates

Highly excited Rydberg states and their interactions play an important role in quantum computing and simulation. These properties can be predicted accurately for alkali atoms with simple Rydberg level structures. However, an extension of these methods to more complex atoms such as alkaline-earth atoms has not been demonstrated or experimentally validated. Here, we present multichannel quantum defect (MQDT) models for highly excited $^{174}$Yb and $^{171}$Yb Rydberg states with $L \leq 2$. The models are developed using a combination of existing literature data and new, high-precision laser and microwave spectroscopy in an atomic beam, and validated by detailed comparison with experimentally measured Stark shifts and magnetic moments. We then use these models to compute interaction potentials between two Yb atoms, and find excellent agreement with direct measurements in an optical tweezer array. From the computed interaction potential, we identify an anomalous F\"orster resonance that likely degraded the fidelity of previous entangling gates in $^{171}$Yb using $F=3/2$ Rydberg states. We then identify a more suitable $F=1/2$ state, and achieve a state-of-the-art controlled-Z gate fidelity of $F=0.994(1)$, with the remaining error fully explained by known sources. This work establishes a solid foundation for the continued development of quantum computing, simulation and entanglement-enhanced metrology with Yb neutral atom arrays.