Endomorphism fields of abelian varieties

We give a sharp divisibility bound, in terms of g, for the degree of the field extension required to realize the endomorphisms of an abelian variety of dimension g over an arbitrary number field; this refines a result of Silverberg. This follows from a stronger result giving the same bound for the order of the component group of the Sato-Tate group of the abelian variety, which had been proved for abelian surfaces by Fite-Kedlaya-Rotger-Sutherland. The proof uses Minkowski's reduction method, but with some care required in the extremal cases when p equals 2 or a Fermat prime.