Anomalous and Anisotropic Nonlinear Susceptibility in the Proximate Kitaev Magnet $\alpha$-RuCl$_3$

The leading order nonlinear (NL) susceptibility, $\chi_3$, in a paramagnet is negative and diverges as $T \rightarrow 0$. This divergence is destroyed when spins correlate and the NL response provides unique insights into magnetic order. Dimensionality, exchange interaction, and preponderance of quantum effects all imprint their signatures in the NL magnetic response. Here, we study the NL susceptibilities in the proximate Kitaev magnet $\alpha$-RuCl$_3$ which differs from the expected antiferromagnetic behavior. For $T< T_c$ = 7.5 K and field $B$ in the ab-plane, we obtain contrasting NL responses in low ($<$ 2 ${T}$) and high field regions. For low fields the NL behavior is dominated by a quadratic response (positive $\chi_2$), which shows a rapid rise below $T_c$. This large $\chi_2 >0$ implies a broken sublattice symmetry of magnetic order at low temperatures. Classical Monte Carlo (CMC) simulations in the standard ${K-H-\Gamma}$ model secure such a quadratic ${B}$ dependence of ${M}$, only for ${T}$ $\approx$ ${T}_c$ with ${\chi}_2$ being zero as ${T}$ $\rightarrow$ 0. It is also zero for all temperatures in exact diagonalization calculations. On the other hand, we find an exclusive cubic term (${\chi}_3$) describes the high field NL behavior well. ${\chi}_3$ is large and positive both below and above ${T}_c$ crossing zero only for ${T}$ $>$ 50 K. In contrast, for $B$~$\parallel$~c-axis, no separate low/high field behaviors is measured and only a much smaller ${\chi}_3$ is apparent.