Exotic quantum phase transitions in a Bose-Einstein condensate coupled to an optical cavity

A new extended Dicke model, which includes atom-atom interactions and a driving classical laser field, is established for a Bose-Einstein condensate inside an ultrahigh-finesse optical cavity. A feasible experimental setup with a strong atom-field coupling is proposed, where most parameters are easily controllable and thus the predicted second-order superradiant-normal phase transition may be detected by measuring the ground-state atomic population. More intriguingly, a novel second-order phase transition from the superradiant phase to the \textquotedblleft Mott" phase is also revealed. In addition, a rich and exotic phase diagram is presented.