Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pairwise commuting angle matrices are exactly what makes rotary position embeddings offset-robust.

desk verdict Clean theorem, modest gains, but the ImageNet attribution to commutativity overreaches. read the letter →

arxiv 2506.03737 v1 pith:YLB74XUM submitted 2025-06-04 cs.CV cs.AI

Hao Yu , Tangyu Jiang , Shuning Jia , Shannan Yan , Shunning Liu , Haolong Qian , Guanghao Li , Shuting Dong
show 2 more authors
Huaisong Zhang Chun Yuan
This is my paper
classification cs.CVcs.AI
keywords rotarypositionembeddingtrainableanglematricespairwisecommutativityRoPEEquationrelativeencodingpositionalrobustnessImageNetclassificationvisiontransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that rotary position embeddings (RoPE) can be generalized from fixed, hand-designed 2D rotations to trainable, larger rotation matrices without losing the property that makes RoPE useful: attention should depend only on the relative position of two tokens, not on where they are anchored. The authors prove that for rotations written as $R(x;A)=\exp(\sum_i A_i x_i)$ with skew-symmetric $A_i$, the product $R(x)^{\top}R(y)$ equals $R(y-x)$ for all positions if and only if the matrices $A_i$ commute pairwise. They then give two trainable constructions that guarantee this commutativity, ComRoPE-AP and ComRoPE-LD, and report that ComRoPE-LD improves over the previous trainable RoPE method, LieRE, by 1.6% at training resolution and 2.9% at $512\times512$ on ImageNet-1K, while keeping accuracy stable under coordinate offsets. If the claim is right, exact offset invariance becomes the organizing principle for designing learned positional encodings, and non-commuting trainable rotations are at a structural disadvantage.

What carries the argument

The carrying object is the parameterized rotation matrix $R(x;A)=\exp(\sum_i A_i x_i)$ built from skew-symmetric angle matrices, together with the RoPE Equation that requires the attention similarity $q^{\top} R(x)^{\top}R(y) k$ to be expressible as a function of the relative position $y-x$ alone. The proof machinery is the matrix exponential identity under commutativity: $e^{Ax}e^{By}=e^{Ax+By}$ if and only if $AB=BA$, extended to $N$ matrices by induction. The two trainable constructions are sufficient conditions for pairwise commutativity: axis-partitioned block-diagonal matrices (ComRoPE-AP) and linearly dependent matrices $B_i=\theta_i(P-P^{\top})$ sharing one base rotation (ComRoPE-LD).

What would settle it

Train a non-commuting trainable-angle RoPE (LieRE-style) with the coordinate-offset protocol of Section 4.3.1: if accuracy stops degrading as the offset standard deviation grows, or matches ComRoPE-LD under those offsets, then pairwise commutativity is not necessary for practical offset robustness. A direct algebraic check is also available: for random skew-symmetric $A,B$, verify that $\exp(Ax)^{\top}\exp(By)=\exp(By-Ax)$ holds exactly when $AB=BA$.

Watch

Extended reading notes

Core claim

The central discovery is a characterization: a rotation-matrix positional encoding $R(x;A)=\exp(\sum_i A_i x_i)$ with skew-symmetric angle matrices $A_i$ satisfies the RoPE Equation — meaning $q^{\top} R(x)^{\top}R(y) k = q^{\top} R(y-x) k$, so attention similarity depends only on the offset $y-x$ — exactly when the $A_i$ pairwise commute. The paper states this as a necessary and sufficient condition (Theorem 1), with the RoPE Equation formalized as the definition of offset robustness. On the constructive side, it shows two sufficient families: ComRoPE-AP, where each axis owns disjoint diagonal blocks of rotation parameters, and ComRoPE-LD, where all axis matrices are scalar multiples of a single trainable skew-symmetric base matrix $P-P^{\top}$. Empirically, ComRoPE-LD outperforms prior trainable and fixed RoPE variants on ImageNet-1K, MS COCO object detection, and UCF-101 3D classification, and — unlike the non-commuting LieRE baseline — retains accuracy when all coordinates receive a common random offset.

Load-bearing premise

The argument defines 'robust' to mean exact translation invariance — attention similarity must equal $q^{\top} R(y-x)k$ — and the theorem's necessity is relative to that formal definition; if approximate invariance suffices in practice, strict pairwise commutativity may be stronger than needed, as the paper's own Appendix H concedes.

Editorial extensions

If this is right

  • Any RoPE variant whose rotations are written as $\exp(\sum_i A_i x_i)$ must have pairwise commuting $A_i$ to satisfy the RoPE Equation; non-commuting trainable rotations such as LieRE cannot guarantee offset invariance.
  • With block size $b=2$ and fixed angles, ComRoPE-AP reduces to vanilla RoPE, and with zero angle matrices it reduces to standard attention, so a pretrained model using either can be fine-tuned under ComRoPE without architectural changes.
  • Larger block sizes place the rotation matrices in a larger subgroup of $SO(d)$ and improve ImageNet accuracy, at an extra time cost of $O(ndb^2)$ for the matrix exponential.
  • ComRoPE-LD needs far fewer extra parameters than LieRE — $d(b+N/b)$ versus $Ndb$ — while reporting higher accuracy and better stability under resolution changes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's commutativity condition implies all learned rotations share a common eigenbasis; this suggests ComRoPE effectively learns per-axis frequency-like scalings rather than truly independent rotations, so its expressiveness is bounded in a way that approximate commutation might relax.
  • The position-perturbation trick in Section 3.4.2 can be read as coordinate jitter augmentation that teaches the model to ignore continuous-position noise; combining it with the offset test could serve as a regularizer for near-commuting variants, an extension the paper does not explore.
  • If the matrix-exponential bottleneck described in the appendices is removed, the same commuting-angle parameterization should transfer to language models, where single-axis sequences make commutativity automatic; the paper leaves that full-scale transfer as future work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes ComRoPE, a rotary position embedding variant in which the position-dependent rotation is exp(Σ_i A_i x_i) with trainable skew-symmetric matrices A_i constrained to pairwise commute. The authors formalize RPE and RoPE equations, prove (Theorem 1) that the rotation difference R(x)^T R(y) is a function of y−x iff the A_i pairwise commute, and instantiate this with two sufficient constructions: ComRoPE-AP (axial partition, one active block per axis) and ComRoPE-LD (linearly dependent blocks B_i = θ_i(P−P^T)). Experiments on ImageNet-1K classification, MS COCO detection, UCF-101 3D classification, and CLIP fine-tuning report consistent gains over LieRE, including a 1.6% (relative) gain at 224×224 and 2.9% at 512×512. Ablations cover coordinate-offset robustness, block size, and position perturbation.

Significance. If the claims hold, the paper makes a useful contribution: it gives a clean formal condition under which a trainable rotation-family position encoding remains exactly translation-invariant, and it shows empirically that enforcing this condition can improve ImageNet accuracy and offset robustness. The appendix proof of Theorem 1 is coherent and complete, the offset ablation in §4.3.1 is the right kind of experiment for commutativity, and source code is provided. The main weaknesses are that the headline accuracy gains are not isolated from other design choices by a matched non-commuting control, the results are single-seed with no confidence intervals, and the abstract's 'essential for scalability and robustness' claim goes beyond what the theorem (which is conditional on a definitional exact-invariance equation) establishes. These weaknesses are fixable without changing the core theoretical result.

major comments (4)
  1. [Section 4.1.2 and Figure 5] The 'Significance of commutativity' paragraph attributes ComRoPE-LD's ImageNet gains, including the 1.6% relative gain at 224×224, to commutativity. This is not supported by the experiments as designed: at 224×224 no coordinate offset is applied, so Theorem 1's equivalence (which concerns whether R(x)^T R(y) can be written as a function of y−x) is vacuous for that comparison. The direct test in §4.3.1 is the right experiment, but Figure 5 plots only the LieRE degradation and does not actually show the claimed ComRoPE invariance, since the ComRoPE curves are 'unpainted.' Please add the ComRoPE offset curves (or a table of numbers) and add a matched non-commuting control—for example, LieRE with the same block size, relative scaling, and position perturbation, or a non-commuting variant of ComRoPE-LD—to separate commutativity from the linearly-dependent parameterization (Proposition 3), relative scaling (§3.4.1), and position perturbation (§3.4.2). Appendix H also concedes that strict commutativity restricts expressiveness, so the construction carries independent effects.
  2. [Table 1 and Footnote 2] The main empirical claims are reported without uncertainty quantification, and the 1.6% figure is a relative improvement, which is easy to misread as an absolute accuracy gap. Table 1 contains one run per condition with no error bars or significance tests, yet the text says ComRoPE-LD 'significantly outperforms' LieRE. Footnote 2 defines improvement relative to the baseline, so the 224×224 gap is about one absolute point (65.49 vs. 64.36) and the 512×512 gap is about 1.5 absolute points (55.29 vs. 53.74). The abstract and §4.1.2 should state absolute gaps and either provide multiple seeds with confidence intervals for the main comparisons or soften the significance language.
  3. [Table 1 vs. Appendix Table 10] The state-of-the-art comparison is incomplete in the main table. Table 1 compares APE, vanilla RoPE, LieRE, and ComRoPE, but RoPE-Mixed—a strong 2D-RoPE baseline that the paper itself discusses in §2.2—appears only in the appendix under a different training recipe (Table 10). Since the abstract and §4.1.2 claim to surpass the current state of the art, the main table should either include RoPE-Mixed under the same protocol or the claim should be explicitly restricted to LieRE.
  4. [Definition 4, Section 3.2, Abstract] The abstract and Section 3.2 say pairwise commutativity is 'essential' for scalability and positional robustness, but the formal result is conditional on Definition 4, which postulates exact translation invariance of the attention similarity, and on the parameterization R(x;A)=exp(Σ_i A_i x_i). The theorem does not cover approximate robustness or resolution extrapolation, and Appendix H acknowledges that strict commutativity restricts expressiveness. Please qualify the necessity claim as exact offset invariance within this parameterized family, and discuss whether weaker conditions (e.g., approximate commutativity) might suffice for practical robustness. This is a framing issue, not a flaw in the proof, but it affects the paper's central claim as stated.
minor comments (4)
  1. [Table 3] Table 3 contains malformed entries (e.g., '50.1455.63' and '79.2194.2480.27'), and the APE row reports 93.10 at 224 but 76.70 at 256, which is implausible and needs correction or explanation.
  2. [Section 4.2 and Table 7] Section 4.2 says ComRoPE-LD uses 'nearly half the number of extra parameters' compared with LieRE, but Table 7 gives a ratio of (b+N/b)/(Nb); for N=2,b=8 this is 9/16≈0.56, not nearly half. Please rephrase the claim.
  3. [Equation (11)] In Eq. (11), the notation diag(σΔX_1,...,σΔX_N)^2 is ambiguous; please clarify whether the square applies elementwise to the diagonal and state the units of σ.
  4. [References] Reference [29] duplicates [28] (Raffel et al.), and in §2.2 the citation 'EV A-02 [8], FiT [8]' should be split into separate entries or the shared [8] should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Theorem 1 is a genuine iff result, the constructions are derived sufficient conditions, and the empirical gains are direct measurements rather than fitted predictions.

full rationale

The paper's central derivation is self-contained. Definition 4 sets out the target property that a parameterized RoPE function should satisfy, namely that the attention similarity depends only on the relative position via R_f(y-x); it does not assume commutativity. Theorem 1 then proves, using the matrix-exponential identities in Lemmas 1-3, that within the family R(x;A)=exp(Σ_i A_i x_i) this target property holds if and only if the angle matrices A_i pairwise commute. The 'if' and 'only if' directions are both argued from the algebra of matrix exponentials, so commutativity is concluded rather than assumed. Propositions 2 and 3 are explicit sufficient constructions (axial-partitioned blocks and linearly dependent matrices) whose commutativity follows directly from the stated algebraic conditions. The experimental results are direct measurements of ImageNet accuracy and ablations against LieRE and other baselines; the trainable angle matrices are fitted by training, but no fitted parameter is renamed as a prediction, and the higher-resolution evaluations are genuine generalization tests. The few self-citations in the paper appear only in the related-work enumeration and are not load-bearing for the theoretical or empirical claims. The main caveat is one of scope, not circularity: Theorem 1 characterizes exact translation invariance within the chosen parameterized family, and the paper's interpretation that commutativity explains the ImageNet gains is an empirical attribution that could be confounded by other design choices, but that is a correctness or experimental-design concern rather than a circularity concern.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

Everything outside the theorem's internal proof is empirical: trained angle matrices, block size, and perturbation intensity are fit or tuned on ImageNet, and the robustness claim depends on the definitional choice of exact translation invariance. No invented physical or conceptual entities are introduced.

free parameters (3)
  • perturbation intensity sigma = 1 (default), explored in {0, 0.2, 0.5, 1.0}
    Hyperparameter controlling Gaussian position noise during training; selected by experiments and directly affects the reported robustness and accuracy values (Section 3.4.2, Figure 7).
  • block size b = 8
    Chosen to balance accuracy and computation; larger block sizes improve performance but increase cost, so b was tuned on ImageNet (Section 4.3.2, Figure 6).
  • learned angle-matrix parameters P_j and scaling factors theta_i = Learned on ImageNet-1K; final values not reported
    These trainable matrices are the core of the method. Their learned distribution is analyzed in Appendix F, but the performance claim is empirical rather than derived from the theory.
assumptions (4)
  • domain assumption RoPE should satisfy exact translation invariance R_f(x)^T R_f(y)=R_f(y-x) as stated in Definition 4, Eq. 2.
    The theorem derives conditions for this equation, but the equation itself is a modeling choice postulating that position encoding depends only on relative position.
  • standard math exp(A) is orthogonal for skew-symmetric A, and every rotation matrix has a skew-symmetric logarithm (Proposition 4).
    Standard Lie theory, cited to references [9] and [11]; used to justify the matrix-exponential representation of rotations.
  • standard math e^A e^B = e^(A+B) if and only if A and B commute (Lemma 1).
    Used as the main algebraic step in the proof of Theorem 1.
  • domain assumption Image patch positions are continuous coordinates normalized to the unit square, with the patch center as the aggregation point.
    This scaling choice underlies resolution robustness but is not derived; it is a standard way to treat continuous image coordinates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices." pith.science (2026). https://pith.science/paper/YLB74XUM

@misc{pith2026250603737,
  author       = {Pith},
  title        = {Pith review of: ComRoPE: Scalable and Robust Rotary Position Embedding Parameterized by Trainable Commuting Angle Matrices},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YLB74XUM}},
  note         = {Machine review of arXiv:2506.03737}
}
read the original abstract

The Transformer architecture has revolutionized various regions since it was proposed, and its effectiveness largely depends on the ability to encode positional information. Traditional position encoding methods exhibit significant limitations due to lack of robustness and flexibility of position. Therefore, Rotary Positional Encoding (RoPE) was proposed to alleviate these issues, which integrates positional information by rotating the embeddings in the attention mechanism. However, RoPE requires manually defined rotation matrices with limited transformation space, constraining the model's capacity. In this work, we propose ComRoPE, which generalizes RoPE by defining it in terms of trainable commuting angle matrices. Specifically, we demonstrate that pairwise commutativity of these matrices is essential for RoPE to achieve scalability and positional robustness. We formally define the RoPE Equation, which is an essential condition that ensures consistent performance with position offsets. Based on the theoretical analysis, we present two types of trainable commuting angle matrices as sufficient solutions to the RoPE equation, which significantly improve performance, surpassing the current state-of-the-art method by 1.6% at training resolution and 2.9% at higher resolution on the ImageNet-1K dataset. Furthermore, our framework shows versatility in generalizing to existing RoPE formulations and offering new insights for future positional encoding research. To ensure reproducibility, the source code and instructions are available at https://github.com/Longin-Yu/ComRoPE

Figures

Figures reproduced from arXiv: 2506.03737 by the authors.

Figure 1
Figure 1. Overview of ComRoPE. Features are arranged into several blocks, each representing a distinct point in the feature space. The [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Different patch sizes result in different relative relation [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Accuracy on ImageNet-1K for various positional encod [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Results over the whole training procedure. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Effect of coordinate offset on LieRE. As the standard [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Accuracy on ImageNet for various block sizes. Larger [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 8
Figure 8. Figure 8: Density distribution of elements in the upper triangular sections of angle matrices from 2D classification experiments. Subfigures [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Robust Filter Attention: Self-Attention as Precision-Weighted State Estimation

    cs.LG 2025-09 unverdicted novelty 7.0 of 10

    Robust Filter Attention models self-attention as consistency-based state estimation under a linear SDE for token trajectories, matching standard attention complexity while showing lower perplexity and better zero-shot...

Reference graph

Works this paper leans on

45 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    Comput- ing the matrix exponential with an optimized taylor polyno- mial approximation.Mathematics, 7:1174, 2019

    Philipp Bader, Sergio Blanes, and Fernando Casas. Comput- ing the matrix exponential with an optimized taylor polyno- mial approximation.Mathematics, 7:1174, 2019. 4

    work page 2019

  2. [2]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020. 1

    arXiv 2005

  3. [3]

    Gim: A million-scale benchmark for genera- tive image manipulation detection and localization.arXiv preprint arXiv:2406.16531, 2024

    Yirui Chen, Xudong Huang, Quan Zhang, Wei Li, Mingjian Zhu, Qiangyu Yan, Simiao Li, Hanting Chen, Hailin Hu, Jie Yang, et al. Gim: A million-scale benchmark for genera- tive image manipulation detection and localization.arXiv preprint arXiv:2406.16531, 2024. 1

    arXiv 2024

  4. [4]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023.URL https://lmsys

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023.URL https://lmsys. org/blog/2023-03-30-vicuna, 3(5),

    2023

  5. [5]

    Li, and Li Fei-Fei

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, K. Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database.2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009. 6

    work page 2009

  6. [6]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. InNorth American Chapter of the Association for Computational Linguistics,

  7. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 1

    arXiv 2010

  8. [8]

    Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171,

    Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xin- long Wang, and Yue Cao. Eva-02: A visual representation for neon genesis.Image and Vision Computing, 149:105171,

Show all 45 references
  1. [9]

    Jean H. Gallier. Basics of classical lie groups: The exponen- tial map, lie groups, and lie algebras. 2001. 3

    2001

  2. [10]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools.arXiv preprint arXiv:2406.12793, 2024. 1

    2024 arXiv

  3. [11]

    Larry C. Grove. Classical groups and geometric algebra

  4. [12]

    Rotary position embedding for vision transformer

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. arXiv preprint arXiv:2403.13298, 2024. 2

    2024 arXiv

  5. [13]

    Translational equivariance in kernel- izable attention.arXiv preprint arXiv:2102.07680, 2021

    Max Horn, Kumar Shridhar, Elrich Groenewald, and Philipp FM Baumann. Translational equivariance in kernel- izable attention.arXiv preprint arXiv:2102.07680, 2021. 2

    2021 arXiv

  6. [14]

    The impact of positional encoding on length generalization in transform- ers.Advances in Neural Information Processing Systems, 36: 24892–24928, 2023

    Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Nate- san Ramamurthy, Payel Das, and Siva Reddy. The impact of positional encoding on length generalization in transform- ers.Advances in Neural Information Processing Systems, 36: 24892–24928, 2023. 1

    2023

  7. [15]

    Autowebglm: A large language model-based web navigating agent

    Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yux- uan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model-based web navigating agent. InProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Dat...

    2024

  8. [16]

    Llava-st: A multimodal large language model for fine-grained spatial- temporal understanding.arXiv preprint arXiv:2501.08282,

    Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao, Xiaoming Wei, and Si Liu. Llava-st: A multimodal large language model for fine-grained spatial- temporal understanding.arXiv preprint arXiv:2501.08282,

    arXiv

  9. [17]

    Lawrence Zitnick, and Piotr Doll ´ar

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Doll ´ar. Microsoft coco: Common objects in context.arXiv preprint arXiv: 1405.0312, 2014. 7

    2014 arXiv

  10. [18]

    We- bglm: towards an efficient web-enhanced question answer- ing system with human preferences

    Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. We- bglm: towards an efficient web-enhanced question answer- ing system with human preferences. InProceedings of the 29th ACM SIGKDD conference on knowledge discovery and da...

    2023

  11. [19]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. Agentbench: Evaluating llms as agents. InThe Twelfth International Conference on Learning Representa- tions, 2024. 1

    2024

  12. [20]

    Swin transformer: Hierarchical vision transformer using shifted windows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021. 2

    2021

  13. [21]

    Relative positional encoding for transformers with linear complexity

    Antoine Liutkus, Ond ˇrej Cıfka, Shih-Lun Wu, Umut Sim- sekli, Yi-Hsuan Yang, and Gael Richard. Relative positional encoding for transformers with linear complexity. InInterna- tional Conference on Machine Learning, pages 7067–7079. PMLR, 2021. 2

    2021

  14. [22]

    Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. InProceed- ings of the IEEE/CVF Conference on Computer Vision a...

    2024

  15. [23]

    Liere: Generalizing rotary position encodings.arXiv preprint arXiv:2406.10322,

    Sophie Ostmeier, Brian Axelrod, Michael E Moseley, Ak- shay Chaudhari, and Curtis Langlotz. Liere: Generalizing rotary position encodings.arXiv preprint arXiv:2406.10322,

    arXiv

  16. [24]

    Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021

    Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation.arXiv preprint arXiv:2108.12409, 2021. 2

    2021 arXiv

  17. [25]

    Improving language understanding by generative pre-training

    A Radford. Improving language understanding by generative pre-training. 2018. 1

    2018

  18. [26]

    Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners.OpenAI blog, 1(8):9, 2019. 1

    2019

  19. [27]

    Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and I

    Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and I. Sutskever. Learning transferable visual models from nat- ural language supervision.International Conference ...

    2021

  20. [29]

    Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer.Journal of machine learning research, 21(140):1–67, 2020. 2

    2020

  21. [30]

    Self- attention with relative position representations.arXiv preprint arXiv:1803.02155, 2018

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self- attention with relative position representations.arXiv preprint arXiv:1803.02155, 2018. 2

    2018 arXiv

  22. [31]

    Ucf101: A dataset of 101 human actions classes from videos in the wild.arXiv preprint arXiv: 1212.0402, 2012

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild.arXiv preprint arXiv: 1212.0402, 2012. 3

    2012 arXiv

  23. [32]

    Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

    Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding.Neurocomputing, 568:127063,

  24. [33]

    End- to-end memory networks.Advances in neural information processing systems, 28, 2015

    Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End- to-end memory networks.Advances in neural information processing systems, 28, 2015. 2

    2015

  25. [34]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  26. [35]

    Attention is all you need.Advances in Neural Information Processing Systems, 2017

    A Vaswani. Attention is all you need.Advances in Neural Information Processing Systems, 2017. 1, 2

    2017

  27. [36]

    Rethinking and improving relative posi- tion encoding for vision transformer

    Kan Wu, Houwen Peng, Minghao Chen, Jianlong Fu, and Hongyang Chao. Rethinking and improving relative posi- tion encoding for vision transformer. InProceedings of the IEEE/CVF International Conference on Computer Vision, pages 10033–10041, 2021. 2

    2021

  28. [37]

    Vit-comer: Vision transformer with convolu- tional multi-scale feature interaction for dense predictions

    Chunlong Xia, Xinliang Wang, Feng Lv, Xin Hao, and Yifeng Shi. Vit-comer: Vision transformer with convolu- tional multi-scale feature interaction for dense predictions. Computer Vision and Pattern Recognition, 2024. 7

    2024

  29. [38]

    Can mllms guide weakly- supervised temporal action localization tasks?arXiv preprint arXiv:2411.08466, 2024

    Quan Zhang and Yuxin Qi. Can mllms guide weakly- supervised temporal action localization tasks?arXiv preprint arXiv:2411.08466, 2024. 1

    2024 arXiv

  30. [39]

    Distilling semantic priors from sam to efficient image restoration models

    Quan Zhang, Xiaoyu Liu, Wei Li, Hanting Chen, Jun- chao Liu, Jie Hu, Zhiwei Xiong, Chun Yuan, and Yunhe Wang. Distilling semantic priors from sam to efficient image restoration models. InProceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages...

    2024

  31. [40]

    IMDPrompter: Adapting SAM to image manipulation detection by cross-view automated prompt learning

    Quan Zhang, Yuxin Qi, Xi Tang, Jinwei Fang, Xi Lin, Ke Zhang, and Chun Yuan. IMDPrompter: Adapting SAM to image manipulation detection by cross-view automated prompt learning. InThe Thirteenth International Conference on Learning Representations, 2025

    2025

  32. [41]

    Rethinking pseudo-label guided learning for weakly supervised temporal action localization from the perspective of noise correction.arXiv preprint arXiv:2501.11124, 2025

    Quan Zhang, Yuxin Qi, Xi Tang, Rui Yuan, Xi Lin, Ke Zhang, and Chun Yuan. Rethinking pseudo-label guided learning for weakly supervised temporal action localization from the perspective of noise correction.arXiv preprint arXiv:2501.11124, 2025. 1 ComRoPE: Scalable and Robust R...

    2025 arXiv

  33. [42]

    (13) SubstitutingA,BwithAx,By, we obtain: eAx+By =e AxeBy

    Necessity (⇒).By the definition ofe A, we have: eA+B = ∞X n=0 (A+B) n n! = ∞X n=0 Pn k=0 n k AkBn−k n! = ∞X n=0 nX k=0 AkBn−k k!(n−k)! = ∞X k=0 Ak k! ! ∞X m=0 Bm m! ! =e AeB. (13) SubstitutingA,BwithAx,By, we obtain: eAx+By =e AxeBy. (14)

  34. [43]

    (16) Lett 2f(t)be the difference between the two expressions above

    Sufficiency (⇐).We have: eAteBt = ∞X n=0 tnAn n! ! ∞X m=0 tmBm m! ! =I+t(A+B) +t 2 · A2 + 2AB+B 2 4 +o(t 2), (15) and e(A+B)t = ∞X n=0 ((A+B)t) n n! =I+t(A+B) +t 2 · (A+B) 2 4 +o(t 2). (16) Lett 2f(t)be the difference between the two expressions above. Thus, we obtain: f(t) = ...

  35. [44]

    (21) SinceA 1,A 2,

    Necessity (⇒).Assuming: eA1x1 eA2x2 · · ·eAkxk =e A1x1+A2x2+···+Akxk , (20) we splitA 1x1 +A 2x2 +· · ·+Ak+1xk+1 into two parts: A1x1 +A 2x2 +· · ·+Ak+1xk+1 = (A1x1 +A 2x2 +· · ·+Akxk) + (Ak+1xk+1). (21) SinceA 1,A 2, . . . ,Ak+1 commute in pairs,A 1x1 + A2x2 +· · ·+Akxk andA ...

  36. [45]

    Then: eA1x1 eA2x2 · · ·eAkxk =e A1x1+A2x2+···+Akxk , (23) implying thatA 1,A 2,

    Sufficiency (⇐).Letx k+1 = 0. Then: eA1x1 eA2x2 · · ·eAkxk =e A1x1+A2x2+···+Akxk , (23) implying thatA 1,A 2, . . . ,Ak commute in pairs. Position Encoding Method Perturbation Intensity Evaluation Resolution 112 128 192 224 256 320 384 448 512 APE 1 48.10 55.25 76.50 93.10 76....

  37. [46]

    We obtain: f(x 1 +y 1, x2 +y 2,

    Sufficiency (⇐).From Equation 31, letx k be replaced withx k +y k andy k with0. We obtain: f(x 1 +y 1, x2 +y 2, . . . , xm +y m) =e A1(x1+y1)+···+Am(xm+ym)eA1·0+···+Am·0 =e A1(x1+y1)+···+Am(xm+ym). (27) Comparing this with Equation equation 31, we get: eA1x1+A2x2+···+AmxmeA1y1...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.