Independence of $\ell$ for Frobenius conjugacy classes attached to abelian varieties

Let $A$ be an abelian variety over a number field $\mathrm E\subset \mathbb C$ and let $\mathbf G$ denote the Mumford--Tate group of $A$. After replacing $\mathrm E$ by a finite extension, the action of the absolute Galois group $\mathrm{Gal}(\overline{\mathrm E}/\mathrm E)$ on the $\ell$-adic cohomology $\mathrm{H}^1_{\mathrm{\acute{e}t}}(A_{\overline{\mathrm E}},\mathbb Q_\ell)$ factors through $\mathbf G(\mathbb Q_\ell).$ We show that for $v$ an odd prime of $\mathrm E$ where $A$ has good reduction, the conjugacy class of Frobenius $\mathrm{Frob}_v$ in $\mathbf G(\mathbb Q_\ell)$ is independent of $\ell$. Along the way we prove that every point in the $\mu$-ordinary locus of the special fiber of Shimura varieties has a special point lifting it.