Enhancing the bandwidth of gravitational-wave detectors with unstable optomechanical filters

For gravitational-wave interferometric detectors, there is a tradeoff between the detector bandwidth and peak sensitivity when focusing on the shot noise level. This has to do with the frequency-dependent propagation phase lag (positive dispersion) of the signal. We consider embedding an active unstable filter---a cavity-assisted optomechanical device operating in the instability regime---inside the interferometer to compensate the phase, and using feedback control to stabilize the entire system. We show that this scheme in principle can enhance the bandwidth without sacrificing the peak sensitivity. However, there is one practical difficulty for implementing it due to the thermal fluctuation of the mechanical oscillator in the optomechanical filter, which puts a very stringent requirement on the environmental temperature and the mechanical quality factor.