That is, ReGLU model takes over and converges faster than ReGLU model

At later stage when the minimum eigenvalue λmin dominates the training process, for sufficiently largek, it holds that Eθ[Lk − ˜Lk]>0