Gravitational Loss-Cone Instability in Stellar Systems with Retrograde Orbit Precession

We study spherical and disk clusters in a near-Keplerian potential of galactic centers or massive black holes. In such a potential orbit precession is commonly retrograde, i.e. direction of the orbit precession is opposite to the orbital motion. It is assumed that stellar systems consist of nearly radial orbits. We show that if there is a loss cone at low angular momentum (e.g., due to consumption of stars by a black hole), an instability similar to loss-cone instability in plasma may occur. The gravitational loss-cone instability is expected to enhance black hole feeding rates. For spherical systems, the instability is possible for the number of spherical harmonics $l \ge 3$. If there is some amount of counter-rotating stars in flattened systems, they generally exhibit the instability independently of azimuthal number $m$. The results are compared with those obtained recently by Tremaine for distribution functions monotonically increasing with angular momentum. The analysis is based on simple characteristic equations describing small perturbations in a disk or a sphere of stellar orbits highly elongated in radius. These characteristic equations are derived from the linearized Vlasov equations (combining the collisionless Boltzmann kinetic equation and the Poisson equation), using the action-angle variables. We use two techniques for analyzing the characteristic equations: the first one is based on preliminary finding of neutral modes, and the second one employs a counterpart of the plasma Penrose-Nyquist criterion for disk and spherical gravitational systems.