Generalized 3G theorem and application to relativistic stable process on non-smooth open sets

Let G(x,y) and G_D(x,y) be the Green functions of rotationally invariant symmetric \alpha-stable process in R^d and in an open set D respectively, where 0<\alpha < 2. The inequality G_D(x,y)G_D(y,z)/G_D(x,z) \le c(G(x,y)+G(y,z)) is a very useful tool in studying (local) Schrodinger operators. When the above inequality is true with a constant c=c(D)>0, then we say that the 3G theorem holds in D. In this paper, we establish a generalized version of 3G theorem when D is a bounded \kappa-fat open set, which includes a bounded John domain. The 3G we consider is of the form G_D(x,y)G_D(z,w)/G_D(x,w), where y may be different from z. When y=z, we recover the usual 3G. The 3G form G_D(x,y)G_D(z,w)/G_D(x,w) appears in non-local Schrodinger operator theory. Using our generalized 3G theorem, we give a concrete class of functions belonging to the non-local Kato class, introduced by Chen and Song, on \kappa-fat open sets. As an application, we discuss relativistic \alpha-stable processes (relativistic Hamiltonian when \alpha=1) in \kappa-fat open sets. We identify the Martin boundary and the minimal Martin boundary with the Euclidean boundary for relativistic \alpha-stable processes in \kappa-fat open sets. Furthermore, we show that relative Fatou type theorem is true for relativistic stable processes in \kappa-fat open sets. The main results of this paper hold for a large class of symmetric Markov processes, as are illustrated in the last section of this paper. We also discuss the generalized 3G theorem for a large class of symmetric stable Levy processes.