Special $p$-groups acting on compact manifolds

Riera proved at arXiv:1412.6964 that the diffeomorphism group of particular compact manifolds are not Jordan by exhibiting subgroups isomorphic to extra-special $p$-groups of exponent $p$ for primes $p$ satisfying some conditions. Generalising the methods of that paper, we construct a compact connected smooth real manifold for every natural number $r$ whose diffeomorphism group contains not only every extra-special $p$-group, but also every special $p$-group of order $p^r$ independently of its exponent for every prime $p$. We obtain a similar statement about finite Heisenberg groups as well as we display a very explicit counterexample to the conjecture of Ghys about Jordan property of diffeomorphism group of compact manifolds.