Coherent population trapping and dynamical instability in the nonlinearly coupled atom-molecule system

We study the possibility of creating a coherent population trapping (CPT) state, involving free atomic and ground molecular condensates, during the process of associating atomic condensate into molecular condensate. We generalize the Bogoliubov approach to this multi-component system and study the collective excitations of the CPT state in the homogeneous limit. We develop a set of analytical criteria based on the relationship among collisions involving atoms and ground molecules, which are found to strongly affect the stability properties of the CPT state, and use it to find the stability diagram and to systematically classify various instabilities in the long-wavelength limit.