Trigonal distortion in the honeycomb iridates: Proximity of zigzag and spiral phases in Na2IrO3

The Kitaev honeycomb model has been suggested as a useful starting point to describe the honeycomb iridates. However, the zigzag ordering seen in Na2IrO3 and the magnetic transition in Li2IrO3 have raised questions to their connection to the Kitaev model and to the microscopic origin of these magnetic states, given their structural similarities. Here we study how the magnetic phases near the Kitaev spin liquid are affected by the inclusion of trigonal distortion of the oxygen octahedra within a purely nearest neighbour model. Using a combination classical analysis and exact diagonalization we show that near the ferromagnetic Kitaev limit a small amount of trigonal distortion, as found in Na2IrO3, stabilizes a zigzag phase. Decreasing the distortion destabilizes the zigzag phase toward a spiral phase that may be relevant for Li2IrO3. Using semi-classical spin-wave calculations we show that this regime is qualitatively consistent with experimentally known features of the dynamical structure factor in Na2IrO3 and speculate on implications for Li2IrO3.