Soft spin-amplitude fluctuations in a Mott-insulating ruthenate

Magnetism in transition-metal compounds (TMCs) has traditionally been associated with spin degrees of freedom, because the orbital magnetic moments are typically largely quenched. On the other hand, magnetic order in 4f- and 5d-electron systems arises from spin and orbital moments that are rigidly tied together by the large intra-atomic spin-orbit coupling (SOC). Using inelastic neutron scattering on the archetypal 4d-electron Mott insulator Ca$_2$RuO$_4$, we report a novel form of excitonic magnetism in the intermediate-strength regime of the SOC. The magnetic order is characterized by ``soft'' magnetic moments with large amplitude fluctuations manifested by an intense, low-energy excitonic mode analogous to the Higgs mode in particle physics. This mode heralds a proximate quantum critical point separating the soft magnetic order driven by the superexchange interaction from a quantum-paramagnetic state driven by the SOC. We further show that this quantum critical point can be tuned by lattice distortions, and hence may be accessible in epitaxial thin-film structures. The unconventional spin-orbital-lattice dynamics in Ca$_2$RuO$_4$ identifies the SOC as a novel source of quantum criticality in TMCs.