The one-loop vacuum energy and RG flow induced by double-trace operators in AdS/CFT and dS/CFT correspondence

We calculate the difference of one-loop vacuum energies for massive scalar field in five-dimensional AdS black hole. (The same is done in five-dimensional deSitter space). In each case this difference is specified by the boundary conditions corresponding to the double-trace operator (massive term) and it describes RG flow in the manner discussed by Gubser-Mitra for pure AdS space. For AdS black hole there occurs instability which is the manifestation of the Hawking-Page phase transition. For stable phase of AdS black hole as well as for deSitter bulk, c-function found beyond the leading order approximation shows the monotonic behaviour consistent with c-theorem.