On swapping the states of two qudits

The SWAP gate has become an integral feature of quantum circuit architectures and is designed to permute the states of two qubits through the use of the well-known controlled-NOT gate. We consider the question of whether a two-qudit quantum circuit composed entirely from instances of the generalised controlled-NOT gate can be constructed to permute the states of two qudits. Arguing via the signature of a permutation, we demonstrate the impossibility of such circuits for dimensions $d \equiv 3$ (mod 4).