REVIEW 4 cited by
Clustering in Causal Attention Masking
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
This work presents a modification of the self-attention dynamics proposed by Geshkovski et al. (arXiv:2312.10794) to better reflect the practically relevant, causally masked attention used in transformer architectures for generative AI. This modification translates into an interacting particle system that cannot be interpreted as a mean-field gradient flow. Despite this loss of structure, we significantly strengthen the results of Geshkovski et al. (arXiv:2312.10794) in this context: While previous rigorous results focused on cases where all three matrices (Key, Query, and Value) were scaled identities, we prove asymptotic convergence to a single cluster for arbitrary key-query matrices and a value matrix equal to the identity. Additionally, we establish a connection to the classical R\'enyi parking problem from combinatorial geometry to make initial theoretical steps towards demonstrating the existence of meta-stable states.
Forward citations
Cited by 4 Pith papers
-
Self-Attention Dynamics with Rotary Position Embeddings: Twisted States and Explicit Consensus Rates on the Sphere
Normalized query/key-only RoPE attention on the sphere has reversible consensus kernels with exact Bessel-aliasing spectra, explicit regional contraction rates from a sharp softmax floor, and RoPE-selected twisted equ...
-
Critical attention scaling in long-context transformers
In a simplified attention model with normalized tokens, the phase boundary between token collapse and identity attention occurs when the attention-temperature scaling factor β_n is of order log n, with constant 1/(1−ρ).
-
Multispin Physics of AI Tipping Points and Hallucinations
A closed-form formula predicts the iteration at which a simplified attention head tips from good to bad output, determined only by token embedding dot products.
-
OT-Transformer: A Continuous-time Transformer Architecture with Optimal Transport Regularization
OT-Transformer replaces a discrete transformer stack with a single ODE and adds a kinetic energy penalty, reporting accuracy gains on four benchmarks.
Discussion (0). Continue with ORCID to comment.