Birkhoff's polytope and unistochastic matrices, N=3 and N=4

The set of bistochastic or doubly stochastic N by N matrices form a convex set called Birkhoff's polytope, that we describe in some detail. Our problem is to characterize the set of unistochastic matrices as a subset of Birkhoff's polytope. For N=3 we present fairly complete results. For N=4 partial results are obtained. An interesting difference between the two cases is that there is a ball of unistochastic matrices around the van der Waerden matrix for N=3, while this is not the case for N=4.