Mott insulating phases and quantum phase transitions of interacting spin-3/2 fermionic cold atoms in optical lattices at half filling

We study various Mott insulating phases of interacting spin-3/2 fermionic ultracold atoms in two-dimensional square optical lattices at half filling. Using a generalized one-band Hubbard model with hidden SO(5) symmetry, we identify two distinct symmetry breaking phases: the degenerate antiferromagnetic spin-dipole/spin-octupole ordering and spin-quadrupole ordering, depending on the sign of the spin-dependent interaction. These two competing orders exhibit very different symmetry properties, low energy excitations and topological characterizations. Near the SU(4) symmetric point, a quantum critical state with a $\pi $-flux phase may emerge due to strong quantum fluctuations, leading to spin algebraic correlations and gapless excitations.