Not yet reviewed by Pith; the record is open.

In this work, we construct three coupled optical modes systems to obtain effective Hamiltonian mediated by coherent dissipative coupling during adiabatic elimination of large dissipation mode. We investigate the cooperative effect of coherent and dissipative photon-photon couplings in an open cavity system, which leads to nonreciprocity with a considerably large isolation ratio and flexible controllability. We discover unidirectional invisibility for electromagnetic wave propagation, which appears at the zero-damping condition (ZDC) for hybrid photon-photon modes and obtain transmission spectrum on the ZDC. We study the influences of the parameters on the nonreciprocal transmission of the system to capture the generic physics of the interference between coherent and dissipative couplings, which accurately reproduces the results of numerical simulation over a broad range of parameters. Moreover, we extend the study of nonreciprocal transmission with the Markovian approximation to the non-Markovian environments, which consist of a collection of oscillators (bosonic photonic modes) and give the adiabatic elimination method with non-Markovian effects. We illustrate that nonreciprocal transmission on ZDC exhibits a crossover from the non-Markovian to the Markovian regimes by controlling the environmental spectral width. This indicates a promising way to enhance or steer quantum nonreciprocal devices in optical cavities and provides potential applications for precision measurements and optical communications with non-Markovian effects.