A new adjoint-sensitivity analysis shows that a normalized influence density evolves exactly along gradient flow and that primacy, recency, and lost-in-the-middle arise from distinct channels under checkable conditions.
A First-Order Mean Field Control Analysis of Transformer Layers under Cross-Entropy Training
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
We study Transformer-type residual layers under cross-entropy training through a continuous-depth mean field control viewpoint. Depth is treated as time, layer parameters as controls, and the residual Transformer recursion as an explicit Euler scheme for a controlled hidden-state flow. For fixed controls, we prove an $O(\varepsilon)$ pathwise approximation of finite-depth trajectories by the continuous flow and combine this with high-probability sampling bounds for the empirical cross-entropy risk. We formulate the limiting population problem as a first-order transport control problem for the law of hidden states and derive a Pontryagin condition whose terminal adjoint contains the softmax residual. We also give finite-class and metric-entropy uniform estimates, compare optimal values, and discuss existence, stability, continuous-to-discrete recovery, initialization, and range estimates for continuous minimizers.
fields
stat.ML 1years
2026 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
An Adjoint-Sensitivity Framework for Lost-in-the-Middle Phenomena in Causal Residual Transformers
A new adjoint-sensitivity analysis shows that a normalized influence density evolves exactly along gradient flow and that primacy, recency, and lost-in-the-middle arise from distinct channels under checkable conditions.