Markov processes with jump kernels decaying at the boundary

The goal of this work is to develop a general theory for non-local singular operators of the type $$ L^{\mathcal{B}}_{\alpha}f(x)=\lim_{\epsilon\to 0} \int_{D,\, |y-x|>\epsilon}\big(f(y)-f(x)\big) \mathcal{B}(x,y)|x-y|^{-d-\alpha}\,dy, $$ and $$ L f(x)=L^{\mathcal{B}}_{\alpha}f(x) - \kappa(x) f(x), $$ in case $D$ is a $C^{1,1}$ open set in $\mathbb{R}^d$, $d\ge 2$. The function $\mathcal{B}(x,y)$ above may vanish at the boundary of $D$, and the killing potential $\kappa$ may be subcritical or critical. From a probabilistic point of view we study the reflected process on the closure $\overline{D}$ with infinitesimal generator $L^{\mathcal{B}}_{\alpha}$, and its part process on $D$ obtained by either killing at the boundary $\partial D$, or by killing via the killing potential $\kappa(x)$. The general theory developed in this work (i) contains subordinate killed stable processes in $C^{1,1}$ open sets as a special case, (ii) covers the case when $\mathcal{B}(x,y)$ is bounded between two positive constants and is well approximated by certain H\"older continuous functions, and (iii) extends the main results known for the half-space in $\mathbb{R}^d$. The main results of the work are the boundary Harnack principle and its possible failure, and sharp two-sided Green function estimates. Our results on the boundary Harnack principle completely cover the corresponding earlier results in the case of half-space. Our Green function estimates extend the corresponding earlier estimates in the case of half-space to bounded $C^{1, 1}$ open sets.