Topological semimetal-to-insulator phase transition between noncollinear and noncoplanar multiple-Q states on a square-to-triangular lattice

Noncollinear and noncoplanar magnetic orders lead to unusual electronic structures and transport properties. We here investigate two types of multiple-Q magnetically ordered states and a topological phase transition between them in two dimensions. One is a coplanar but noncollinear double-Q state on a square lattice, which is a semimetal accommodating massless Dirac electrons. The other is a noncoplanar triple-Q state on a triangular lattice, which is a Chern insulator showing the quantum anomalous Hall effect. We discuss the peculiar electronic structures in these two multiple-Q states in a unified way on the basis of the Kondo lattice model, which suggests a quantum phase transition between the two states in a continuous change of lattice geometry between the square and triangular lattices. We systematically examine the possibility of such a transition by using the mean-field approximation for the ground state of the periodic Anderson model. After clarifying the parameter region in which the double-Q (triple-Q) state is stabilized on the square (triangular) lattice, we show that a continuous topological phase transition indeed takes place between the double-Q Dirac semimetal and the triple-Q Chern insulator on the square-to-triangular lattice. The nature of the transition is discussed by the topologically-protected edge states as well as the bulk magnetic and electronic properties. The results indicate that unusual critical phenomena may occur at finite temperature related with multiple-Q chiral spin-liquid states.