Wigner's Theorem for stabilizer states and quantum designs

We describe the symmetry group of the stabilizer polytope for any number $n$ of systems and any prime local dimension $d$. In the qubit case, the symmetry group coincides with the linear and anti-linear Clifford operations. In the case of qudits, the structure is somewhat richer: for $n=1$, it is a wreath product of permutations of bases and permutations of the elements within each basis. For $n>1$, the symmetries are given by affine symplectic similitudes. These are the affine maps that preserve the symplectic form of the underlying discrete phase space up to a non-zero multiplier. We phrase these results with respect to a number of a priori different notions of "symmetry'', including Kadison symmetries (bijections that are compatible with convex combinations), Wigner symmetries (bijections that preserve inner products), and symmetries realized by an action on Hilbert space. Going beyond stabilizer states, we extend an observation of Heinrich and Gross (Ref. [25]) and show that the symmetries of fairly general sets of Hermitian operators are constrained by certain moments. In particular: the symmetries of a set that behaves like a 3-design preserve Jordan products and are therefore realized by conjugation with unitaries or anti-unitaries. (The structure constants of the Jordan algebra are encoded in an order-three tensor, which we connect to the third moments of a design). This generalizes Kadison's formulation of the classic Wigner Theorem on quantum mechanical symmetries.