Pith. sign in

REVIEW 4 cited by

LieRE: Lie Rotational Positional Encodings

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.10322 v5 pith:KPJYBWVQ submitted 2024-06-14 cs.CV cs.LG

Sophie Ostmeier , Brian Axelrod , Maya Varma , Michael E. Moseley , Akshay Chaudhari , Curtis Langlotz This is my paper · ORCID
classification cs.CVcs.LG
keywords encodingslierematricespositionalrelativeropeeffectivenesselements
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Transformer architectures rely on position encodings to model the spatial structure of input data. Rotary Position Encoding (RoPE) is a widely used method in language models that encodes relative positions through fixed, block-diagonal, rotation matrices applied to key-query interactions. We hypothesize that this inductive bias limits their RoPE's effectiveness for modalities with high dimensional structure. Lie Relative Encodings (LieRE) introduce a principled generalization of RoPE, aimed at increasing the representational capacity of positional encodings in transformers. Instead of fixed 2D rotations, LieRE learns dense skew-symmetric matrices (Lie algebra elements), which are then differentiable mapped to form high-dimensional rotation matrices (Lie group elements). This results in richer, learnable, and continuous, encodings of both relative and absolute positional information. We demonstrate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions while maintaining computational efficiency. The code and checkpoints are publicly available at https://github.com/StanfordMIMI/LieRE.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

  2. Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer

    cs.CV 2025-11 conditional novelty 6.0 of 10

    Expressing all RoPE positions on the query's grid ('one attention, one scale') plus a small boundary content-exchange step restores mixed-resolution diffusion generation that naive position interpolation destroys.

  3. SeqPE: Transformer with Sequential Position Encoding

    cs.LG 2025-06 reject novelty 6.0 of 10

    SeqPE encodes each position as a symbolic digit sequence through a small Transformer, and with contrastive plus distillation losses it reports improved extrapolation in language, QA, and image classification.

  4. ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Trainable commuting angle matrices generalize Rotary Position Embedding and improve accuracy and resolution robustness on vision Transformers.

Pith tools