REVIEW 7 cited by
LieRE: Lie Rotational Positional Encodings
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformer architectures rely on position encodings to model the spatial structure of input data. Rotary Position Encoding (RoPE) is a widely used method in language models that encodes relative positions through fixed, block-diagonal, rotation matrices applied to key-query interactions. We hypothesize that this inductive bias limits their RoPE's effectiveness for modalities with high dimensional structure. Lie Relative Encodings (LieRE) introduce a principled generalization of RoPE, aimed at increasing the representational capacity of positional encodings in transformers. Instead of fixed 2D rotations, LieRE learns dense skew-symmetric matrices (Lie algebra elements), which are then differentiable mapped to form high-dimensional rotation matrices (Lie group elements). This results in richer, learnable, and continuous, encodings of both relative and absolute positional information. We demonstrate the effectiveness of LieRE on 2D and 3D vision tasks, showing that it generalizes well to higher input resolutions while maintaining computational efficiency. The code and checkpoints are publicly available at https://github.com/StanfordMIMI/LieRE.
Forward citations
Cited by 7 Pith papers
-
Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation
Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...
-
Power law graph attention: exact generalization of scaled dot-product attention, empirical collapse at inference
After training, a 110M-parameter power-law attention model's learned scoring operator becomes nearly input-invariant, so inference can cache it; the paper proves this collapse conditionally, measures it at 1e-6 and be...
-
Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer
Expressing all RoPE positions on the query's grid ('one attention, one scale') plus a small boundary content-exchange step restores mixed-resolution diffusion generation that naive position interpolation destroys.
-
SeqPE: Transformer with Sequential Position Encoding
SeqPE encodes each position as a symbolic digit sequence through a small Transformer, and with contrastive plus distillation losses it reports improved extrapolation in language, QA, and image classification.
-
ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices
Trainable commuting angle matrices generalize Rotary Position Embedding and improve accuracy and resolution robustness on vision Transformers.
-
Learning the RoPEs: Better 2D and 3D Position Encodings with STRING
STRING parameterizes translation-invariant position encodings as exponentials of commuting skew-symmetric generators, proving they are exactly RoPE in a learnable orthogonal basis, and shows practical gains in 2D/3D v...
-
ENSAM: an efficient foundation model for interactive segmentation of 3D medical images
ENSAM, a 5.5M-parameter promptable 3D segmentation model trained on a 4,471-volume coreset with one GPU for 6 hours, scored 2.404 DSC AUC on the CVPR 2025 hidden test, beating VISTA3D and SAM-Med3D and nearly matching SegVol.
Discussion (0). Continue with ORCID to comment.