Effect of quintic nonlinearity on modulation instability in coupled nonlinear Schroedinger systems

Modulation instability (MI) in continuous media described by a system of two cubic-quintic nonlinear Schr\"odinger equations (NLSE) has been investigated with a focus on revealing the contribution of the quintic nonlinearity to the development of MI in its linear and nonlinear stages. For the linear stage we derive analytic expression for the MI gain spectrum and compare its predictions with numerical simulations of the governing coupled NLSE. It is found that the quintic nonlinearity significantly enhances the growth rate of MI and alters the features of this well known phenomenon by suppressing its time-periodic character. For the nonlinear stage by employing a localized perturbation to the constant background we find that the quintic nonlinearity notably changes the behavior of MI in the central oscillatory region of the integration domain. In numerical experiments we observe emergence of multiple moving coupled solitons if the parameters are in the domain of MI. Possible applications of the obtained results to mixtures of Bose-Einstein condensates and bimodal light propagation in waveguide arrays are discussed.