REVIEW 4 cited by
OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Transformers have achieved state-of-the-art performance in numerous tasks. In this paper, we propose a continuous-time formulation of transformers. Specifically, we consider a dynamical system whose governing equation is parametrized by transformer blocks. We leverage optimal transport theory to regularize the training problem, which enhances stability in training and improves generalization of the resulting model. Moreover, we demonstrate in theory that this regularization is necessary as it promotes uniqueness and regularity of solutions. Our model is flexible in that almost any existing transformer architectures can be adopted to construct the dynamical system with only slight modifications to the existing code. We perform extensive numerical experiments on tasks motivated by natural language processing, image classification, and point cloud classification. Our experimental results show that the proposed method improves the performance of its discrete counterpart and outperforms relevant comparing models.
Forward citations
Cited by 4 Pith papers
-
Deep Delta Learning
Replacing additive residual connections with a gated rank-1 delta update that interpolates identity, projection, and reflection slightly improves language modeling and downstream averages in reported 124M/353M runs.
-
From Self-Attention to Connection Laplacian: A Unified Operator View of Transformers
Multi-head attention is exactly a scaled edge-dependent connection walk, and trained Transformers empirically develop stable walks and approximate scaled-isometric transports that strengthen with scale.
-
Attention's forward pass and Frank-Wolfe
Hardmax self-attention is shown to be a Frank-Wolfe iteration; with positive-definite key-query it converges to Voronoi-cell vertices, and a Markov-chain version of soft attention is metastable there for exponential-i...
-
Physics- and geometry-aware spatio-spectral graph neural operator for time-independent and time-dependent PDEs
A submission whose abstract describes a new graph neural operator for PDEs but whose full text is a different paper, leaving the claimed method and results unverifiable.
Discussion (0). Sign in to comment.