Quantum effects on the dynamics of a two-mode atom-molecule Bose-Einstein condensate

We study the system of coupled atomic and molecular condensates within the two-mode model and beyond mean-field theory (MFT). Large amplitude atom-molecule coherent oscillations are shown to be damped by the rapid growth of fluctuations near the dynamically unstable molecular mode. This result contradicts earlier predictions about the recovery of atom-molecule oscillations in the two-mode limit. The frequency of the damped oscillation is also shown to scale as $\sqrt{N}/\log N$ with the total number of atoms $N$, rather than the expected pure $\sqrt{N}$ scaling. Using a linearized model, we obtain analytical expressions for the initial depletion of the molecular condensate in the vicinity of the instability, and show that the important effect neglected by mean field theory is an initially non-exponential `spontaneous' dissociation into the atomic vacuum. Starting with a small population in the atomic mode, the initial dissociation rate is sensitive to the exact atomic amplitudes, with the fastest (super-exponential) rate observed for the entangled state, formed by spontaneous dissociation.