Moduli spaces of framed symplectic and orthogonal bundles on P2 and the K-theoretic Nekrasov partition functions

Let $K$ be the compact Lie group $USp(N/2)$ or $SO(N, R)$. Let $M^K_n$ be the moduli space of framed K-instantons over $S^4$ with the instanton number $n$. By Donaldson (1984), $M^K_n$ is endowed with a natural scheme structure. It is a Zariski open subset of a GIT quotient of $\mu^{-1}(0)$, where $\mu$ is a holomorphic moment map such that $\mu^{-1}(0)$ consists of the ADHM data. The purpose of the paper is to study the geometric properties of $\mu^{-1}(0)$ and its GIT quotient, such as complete intersection, irreducibility, reducedness and normality. If $K=USp(N/2)$ then $\mu$ is flat and $\mu^{-1}(0)$ is an irreducible normal variety for any $n$ and even $N$. If $K = SO(N, R)$ the similar results are proven for low $n$ and $N$. As an application one can obtain a mathematical interpretation of the K-theoretic Nekrasov partition function of Nekrasov and Shadchin (2004).