Observation of Optical Solitons and Abnormal Modulation Instability in Liquid Crystals with Negative Dielectric Anisotropy

We investigate theoretically and experimentally the optical beam propagation in the nematic liquid crystal with negative dielectric anisotropy, which is aligned homeotropically in a $80\mu m$-thickness planar cell in the presence of an externally voltage. It is predicted that the nonlocal nonlinearity of liquid crystal undergo an oscillatory response function with a negative nonlinear refractive index coefficient. We found that the oscillatory nonlocal nonlinearity can support stable bright solitons, which are observed in experiment. We also found that abnormal modulation instability occurs with infinity gain coefficient at a fixed spatial frequency, which is no depend on the beam intensity. We observed the modulation instability in the liquid crystal at a very low intensity ($0.26W/cm^2$), and the maximum gain frequency were found kept unchange when beam power changes over 2-3 orders of magnitude.