A lower bound for periods of matrices

For a nonsingular integer matrix A, we study the growth of the order of A modulo N. We say that a matrix is exceptional if it is diagonalizable, and a power of the matrix has all eigenvalues equal to powers of a single rational integer, or all eigenvalues are powers of a single unit in a real quadratic field. For exceptional matrices, it is easily seen that there are arbitrarily large values of N for which the order of A modulo N is logarithmically small. In contrast, we show that if the matrix is not exceptional, then the order of A modulo N goes to infinity faster than any constant multiple of log N.