Mode coupling and resonance instabilities in quasi-two-dimensional dust clusters in complex plasmas

Small quasi-two-dimensional (2D) dust clusters consisting of three to eleven particles are formed in an argon plasma under varying rf power. Their normal modes are investigated through their mode spectra obtained from tracking thermal motion of the particles. Detailed coupling patterns between their horizontal and vertical modes are detected for particle numbers up to seven and discrete instabilities are found for dust clusters with particle number greater than nine, as predicted in previous theory on ion-flow induced mode coupling in small clusters. The instabilities are proven to be induced by resonance between coupled horizontal and vertical normal modes.