Competition between three-sublattice order and superfluidity in the quantum 3-state Potts model of ultracold bosons and fermions on a square optical lattice

We study a quantum version of the three-state Potts model that includes as special cases the effective models of bosons and fermions on the square lattice in the Mott insulating limit. It can be viewed as a model of quantum permutations with amplitudes J_parallel and J_perp for identical and different colors, respectively. For J_parallel=J_perp>0, it is equivalent to the SU(3) Heisenberg model, which describes the Mott insulating phase of 3-color fermions, while the parameter range J_perp<min(0,-J_parallel) can be realized in the Mott insulating phase of 3-color bosonic atoms. Using linear flavor wave theory, infinite projected entangled-pair states (iPEPS), and continuous-time quantum Monte-Carlo simulations, we construct the full T=0 phase diagram, and we explore the T>0 properties for J_perp<0. For dominant antiferromagnetic J_parallel interactions, a three-sublattice long-range ordered stripe state is selected out of the ground state manifold of the antiferromagnetic Potts model by quantum fluctuations. Upon increasing |J_perp|, this state is replaced by a uniform superfluid for J_perp<0, and by an exotic three-sublattice superfluid followed by a two-sublattice superfluid for J_perp>0. The transition out of the uniform superfluid (that can be realized with bosons) is shown to be a peculiar type of Kosterlitz-Thouless transition with three types of elementary vortices.