Constraining conformal field theories with a slightly broken higher spin symmetry

We consider three dimensional conformal field theories that have a higher spin symmetry that is slightly broken. The theories have a large N limit, in the sense that the operators separate into single trace and multitrace and obey the usual large N factorization properties. We assume that the spectrum of single trace operators is similar to the one that one gets in the Vasiliev theories. Namely, the only single trace operators are the higher spin currents plus an additional scalar. The anomalous dimensions of the higher spin currents are of order 1/N. Using the slightly broken higher spin symmetry we constrain the three point functions of the theories to leading order in N. We show that there are two families of solutions. One family can be realized as a theory of N fermions with an O(N) Chern-Simons gauge field, the other as a N bosons plus the Chern-Simons gauge field. The family of solutions is parametrized by the 't Hooft coupling. At special parity preserving points we get the critical O(N) models, both the Wilson-Fisher one and the Gross-Neveu one. Our analysis also fixes the on shell three point functions of Vasiliev's theory on AdS_4 or dS_4.