Pith. sign in

REVIEW 3 major objections 6 minor 39 references

The paper proves that any normalized positive-definite attention kernel is, in expectation, a RoPE-style rotation with frequencies sampled from the kernel's own Fourier transform; ClockRoPE uses this to encode daily and weekly user routines

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 17:15 UTC pith:M6F7D7QH

load-bearing objection RFR is a clean but near-tautological adaptation of RFF to rotations; the paper's best variant (folded) is not covered by the theory, yet the empirical story is credible. the 3 major comments →

arxiv 2607.26369 v1 pith:M6F7D7QH submitted 2026-07-29 cs.LG

ClockRoPE: Random Fourier Rotations for Temporal Routine Modeling

Yiwen Chen , Joshua Ainslie , Krzysztof Choromanski , Xiang Gao , Su-Lin Wu , Yiping Yuan , Qian Sun This is my paper
classification cs.LG
keywords rotary position embeddingRoPERandom Fourier Rotationssequential recommendationperiodic attentiontemporal encodingpositive-definite kernelsgenerative retrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper sets out to show that rotary position embeddings are far more expressive than the standard log-linear frequency schedule suggests: any continuous, normalized, positive-definite attention modulation function can be recovered, on average, by sampling rotation frequencies from that function's own Fourier transform. If true, position-encoding design becomes a spectral-sampling problem, and arbitrary distance-correlation patterns—especially periodicity—can be installed without new input features or extra dimensions. The authors instantiate this as ClockRoPE, drawing daily and weekly frequencies from cosine and periodic-Gaussian priors, and report offline ranking gains plus production A/B improvements in valued engagement at a large video-sharing platform. The practical payoff is that RoPE's recency decay and ClockRoPE's routine periodicity run on separate feature halves and compose, giving the best online results.

Core claim

The central claim is Proposition 3.1: if f is continuous, positive-definite, and f(0)=1, then sampling i.i.d. frequencies ξ_j from the Fourier transform τ (which is a probability density) and rotating each query/key feature pair by angle 2πξ_j p gives an estimator whose expectation equals q^T k f(p_m − p_n). The proof lifts the classical random-feature derivation from feature maps to Givens rotations, so the modulation multiplies the dot product without additive masks. For periodic kernels, the spectrum becomes a discrete probability mass over harmonics k/T, enabling ClockRoPE to sample a small set of daily and weekly harmonics from a periodic-Gaussian or cosine prior. The best-performing va

What carries the argument

The central object is the Random Fourier Rotation estimator: block-diagonal Givens rotation matrices with angles 2πξ_j p, where ξ_j are sampled from the Fourier transform τ of the target modulation f. Because τ is a probability density, the expectation of the rotated dot product equals q^T k f(p_m − p_n); for periodic kernels the spectrum collapses to a discrete PMF over integer multiples of 1/T, with probability masses given by the kernel's Fourier coefficients. ClockRoPE instantiates this with cosine and periodic-Gaussian priors and truncates the harmonic series, since Gaussian coefficients decay exponentially in k². The folded variant uses only non-negative frequencies, justified by assum

Load-bearing premise

The best-performing folded variant assumes that, for every feature pair, the query and key vectors are roughly aligned so the rotation's sine component is negligible compared with the cosine component; if learned queries and keys violate this alignment, the folded estimator is no longer a faithful implementation of the intended periodic kernel.

What would settle it

On a trained ClockRoPE-gaussian-fold model, compute for many query-key pairs and feature pairs the ratio |q_{2j}k_{2j+1} − q_{2j+1}k_{2j}| / |q_{2j}k_{2j} + q_{2j+1}k_{2j+1}|; if the median ratio is not much smaller than 1, the sine term is not negligible and the folded model is not faithfully implementing the periodic Gaussian kernel. A controlled synthetic dataset with known periodic ground truth would then reveal systematic bias in the realized attention matrix.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Any normalized continuous positive-definite attention modulation can be encoded multiplicatively in attention, so RoPE-style efficient implementations apply without additive bias terms or quadratic masks.
  • For periodic kernels the frequency distribution is a discrete set of harmonics with computable probabilities, so ClockRoPE needs only a few sampled frequencies and adds no trainable parameters or input features.
  • RoPE's long-term decay and ClockRoPE's periodicity can be applied to disjoint feature halves and their effects compose, matching the paper's offline and online results.
  • Because the estimator is unbiased at initialization and converges exponentially fast in dimension when projection weights start i.i.d., the periodic prior is enforced early in training rather than learned slowly.
  • Production A/B results indicate that modeling time-of-day and day-of-week routines improves valued engagement and reduces serving cost, the empirical payoff of the approach.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same Fourier-sampling recipe likely extends to any stationary correlation pattern beyond periodicity—for example, Matern or rational-quadratic kernels—so a testable next step is replacing log-linear RoPE frequencies with samples from such spectra and measuring length generalization.
  • The folded distribution's asymmetry suggests that temporal causality (more attention to 'same time yesterday' than 'same time tomorrow') can be encoded by breaking sign symmetry; one could test whether flipping the sign convention reverses the observed weekly-boundary attention asymmetry.
  • The learnable-frequency experiments in the paper hint that ClockRoPE's value may be as a high-quality initialization rather than a final kernel; annealing sampled frequencies toward learned values during training could combine prior stability with data-driven adaptation.
  • If the unbiasedness argument carries to multi-dimensional time coordinates (hour-of-day and day-of-week as a vector), ClockRoPE might generalize to multi-periodic kernels naturally, without sharding features by period.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper introduces ClockRoPE, a method for encoding temporal periodicity in transformer attention by sampling rotary position embedding (RoPE) frequencies from the Fourier transform of a positive-definite attention modulation function. The central theoretical result (Proposition 3.1) states that for any normalized continuous positive-definite kernel f, the expectation of the RoPE-style rotation-based score equals the original query-key dot product times f(temporal distance), provided frequencies are drawn from the spectral measure of f. This is proved via Bochner's theorem and the complex representation of 2D rotations. Proposition 3.2 provides a McDiarmid concentration bound. The authors instantiate this framework with cosine and periodic Gaussian priors, and also propose 'folded' distributions that use only non-negative frequencies to induce directional awareness. Experiments on a production-scale generative recommendation system at a major video-sharing platform show consistent offline and online gains, with the Gaussian-folded variant performing best and being deployed. The paper also reports deployment metrics including increased valued engagement and reduced serving cost.

Significance. If the results hold, ClockRoPE offers a principled replacement for the standard log-linear RoPE frequency schedule in domains with periodic structure. The theoretical derivation is transparent and the production A/B results are a notable strength. However, the contribution is largely an application of Random Fourier Features to rotations; the core theorem is essentially Bochner's theorem restated in rotation form. Moreover, the best-performing folded variant falls outside the unbiasedness guarantee, so the link between theory and the deployed method is incomplete. The paper is likely to be of interest to the sequential recommendation and position-encoding communities, but the gap requires attention.

major comments (3)
  1. [§3.3.3, Eq. (24)] The folded frequency distribution does not preserve the unbiasedness of Proposition 3.1. With P_fold, E[cos(2πξδ)] = f(δ), but E[sin(2πξδ)] is generally non-zero and odd in δ. Hence the expected per-pair logit is (q·k)f(δ) + (q×k)g(δ), which equals the intended kernel only if q×k = 0 or g = 0. Since the Gaussian-fold variant is the best performer (Tables 2–4) and is the deployed configuration (§4.2, §4.5), the reported gains may reflect an uncharacterized directional bias rather than the Random Fourier Rotation mechanism. This is load-bearing: the central theoretical guarantee does not apply to the main empirical result.
  2. [§3.3.3] The paper asserts that for 'roughly aligned' queries and keys, |q_{m,2j}k_{n,2j+1} - q_{m,2j+1}k_{n,2j}| << |q_{m,2j}k_{n,2j} + q_{m,2j+1}k_{n,2j+1}|, so the sine term can be ignored. This condition is neither proven nor empirically verified. For random or learned q/k, the cross term generally has the same scale as the dot term. The authors should either provide a theoretical justification, an empirical histogram of the ratio for the trained models, or explicitly characterize the folded variant as an additional inductive bias outside the scope of Proposition 3.1. Without this, the 'mathematically grounded' claim is overstated.
  3. [§3.2, Algorithm 2] Proposition 3.1 guarantees unbiasedness in expectation over the random frequencies, but Algorithm 2 samples frequencies once per layer and fixes them for training and inference. Thus the deployed model uses a single random realization, and Proposition 3.2 is the relevant bound. The manuscript should make this explicit, since otherwise the reader may think the realized attention matrix is itself unbiased. This is a clarity issue that affects the interpretation of the theory.
minor comments (6)
  1. [§3.2, after Eq. (13)] The statement 'when Σ ((q_m^(j))^T k_n^(j))^2 scales linearly with d' does not match the denominator of the bound, which uses Σ (||q^(j)|| ||k^(j)||)^2. Please correct.
  2. [Eq. (1)] There is a stray '{' in the displayed formula; the intended target is likely q_m^T k_n f(p_m - p_n).
  3. [Footnote 2] The statement that the periodic Gaussian 'is positive definite when σ << T/2' is technically inaccurate; the Fourier coefficients α_k are nonnegative for all σ, and the condition σ << T/2 is used for the approximation in Eq. (18). Please rephrase.
  4. [Tables 2–4] No confidence intervals or significance tests are reported for the online A/B metrics; given the small percentage changes, it would be helpful to report statistical significance.
  5. [References] Several references are dated 2026; please ensure they are publicly available or clearly marked as preprints.
  6. [§4.1.1, Arm 2] The description says 'We divide all features into two halves and apply ClockRoPE-cosine-sym with daily period and weekly period to each half respectively.' Please specify how the two halves are allocated (e.g., by contiguous feature indices) and how the sharding interacts with multi-head attention.

Circularity Check

1 steps flagged

RFR theorem restates Bochner's identity by construction; folded variant's theory relies on an unverified alignment condition.

specific steps
  1. self definitional [Section 3.2, Proposition 3.1 and Eqs. (6), (12)]
    "Let ξ0,ξ1,...,ξd/2−1 i.i.d.∼τ(ξ), where τ(ξ) is the Fourier transform of a continuous positive definite kernel f: R→R with f(0)=1... Then the Random Fourier Rotation estimator ... is an unbiased estimator of q⊤m kn f(pm−pn). ... f(pm−pn)=∫R e^{i2πξ(pm−pn)}τ(ξ)dξ=Eξ∼τ(ξ)[e^{i2πξ(pm−pn)}] ... =q⊤m kn f(pm−pn)"

    The sampling distribution τ is defined as the Fourier transform of the target f (Eq. 6); Bochner's theorem then gives E[e^{i2πξδ}]=f(δ). The estimator's expectation in Eq. (12) is exactly this identity summed over feature pairs. Therefore the theorem's conclusion—unbiased approximation of f—is built into the choice of τ. The result is a restatement of Bochner/RFF in rotation form, not an independent derivation; the 'approximation' is the definition.

full rationale

The central theoretical step, Proposition 3.1, is circular in the strict pattern-1 sense: the frequency distribution is defined as the Fourier transform of the modulation function f, so the unbiasedness of the rotation estimator reduces to Bochner's identity by construction (Eq. 6 collapses into Eq. 12). This is mathematically valid and useful as a design rule, but as a 'first-principles result' it does not derive anything beyond its own input. The folded-frequency variant, which is the best offline and deployed configuration (Tables 2–5, §4.5), is not covered by that theorem: under P_fold, E[sin(2πξδ)] is generally nonzero, so the per-pair logit in Eqs. (24)–(25) has mean (q·k)f(δ)+(q×k)g(δ), not the intended kernel. The paper asserts |q×k|≪|q·k| for 'roughly aligned' queries and keys without constraint or evidence; this is a correctness gap rather than a circularity, but it means the reported gains may come from an unmodeled directional bias. The Gaussian width σ and truncation s are also tuned by parameter sweep (§4.3.1), further weakening the first-principles framing. No load-bearing self-citation exists: Bochner and Herglotz are external, and self-citations in related work (e.g., STRING, RFF) are background. The independent offline/online empirical content and the specific ClockRoPE engineering prevent a score of 8 or 10, but the core theoretical 'prediction' reduces by construction, giving 6.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new physical entities are introduced. The method introduces a sampling distribution over frequencies, but that is a mathematical construction, not an entity. The free parameters are standard kernel/hyperparameters, though the truncation and split ratios are chosen by validation.

free parameters (3)
  • Periodic Gaussian width σ = 2.0 hr (best via sweep)
    Controls the width of the periodic Gaussian attention kernel; selected in Table 5 by MAP sweep.
  • Truncation size s = 6 (best via sweep)
    Truncation size for the discrete harmonic PMF; larger values do not improve MAP (Table 5).
  • Feature split ratio = half RoPE / half ClockRoPE; half daily / half weekly
    Architectural allocation of feature channels to different periods and to RoPE; chosen by hand and tested in §4.1.2/§4.3.2.
axioms (5)
  • standard math Bochner's theorem
    Used in Proposition 3.1 to represent pd kernels as expectation over frequency; classical theorem.
  • standard math Herglotz's theorem
    Used in Corollary 3.3 to justify that periodic pd kernels have non-negative Fourier coefficients summing to f(0).
  • domain assumption Periodic Gaussian positive definiteness
    Assumes σ ≪ T/2 so the periodic extension is smooth and Fourier coefficients non-negative; satisfied for σ=2h, T=24h (§3.3.2).
  • ad hoc to paper Folded distribution approximation
    Assumes cross term |q_x k_y - q_y k_x| is negligible vs dot term so the sin component can be ignored; not proven, §3.3.3.
  • domain assumption Softmax modulation heuristic
    Assumes lowering logit upper bounds for out-of-phase pairs concentrates softmax attention on same-phase pairs; empirically observed but not proved.

pith-pipeline@v1.3.0-daily-deepseek · 15863 in / 17648 out tokens · 175130 ms · 2026-08-01T17:15:35.590243+00:00 · methodology

0 comments
read the original abstract

Rotary Position Embedding (RoPE) has been widely adopted in transformer-based large language models. However, its log-linear frequency schedule, originally designed to produce long-term attention decay, limits its adoption in domains with more complex distance-correlation patterns, such as temporal periodicity in sequential recommendation. We investigate the expressiveness of general query/key rotations and find that any normalized continuous positive-definite attention modulation function can be approximated by random rotations induced by its own Fourier transform, which we term Random Fourier Rotations. Building on this theory, we propose ClockRoPE for routine modeling in sequential recommendation, where rotation frequencies are derived from periodic attention modulation functions. In online A/B tests, ClockRoPE demonstrates consistent improvements in valued engagement metrics, and has been successfully deployed in production-scale generative retrieval system at a major video-sharing platform.

Figures

Figures reproduced from arXiv: 2607.26369 by Joshua Ainslie, Krzysztof Choromanski, Qian Sun, Su-Lin Wu, Xiang Gao, Yiping Yuan, Yiwen Chen.

Figure 1
Figure 1. Figure 1: Long-term decay of RoPE 2.3 Fourier Methods in Sequence Modeling Fourier methods are a common tool in sequence modeling. One early example for Transformer models in particular is FNet [19], which replaced the self-attention mechanism in Transformer layers with the discrete Fourier transform. Despite lacking learnable pa￾rameters, the discrete Fourier transform served as an efficient token "mixing" mechanis… view at source ↗
Figure 2
Figure 2. Figure 2: Visualization of average attention probability over [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Visualization of average attention probability [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

39 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    1933.Monotone Funktionen, Stieltjessche Integrale und har- monische Analyse

    Salomon Bochner. 1933.Monotone Funktionen, Stieltjessche Integrale und har- monische Analyse. Springer

    1933

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners.Advances in neural information processing systems33 (2020), 1877–1901

    2020

  3. [3]

    Zheng Chai, Qin Ren, Xijun Xiao, Huizhi Yang, Bo Han, Sijun Zhang, Di Chen, Hui Lu, Wenlin Zhao, Lele Yu, et al . 2025. Longer: Scaling up long sequence modeling in industrial recommenders. InProceedings of the Nineteenth ACM Conference on Recommender Systems. 247–256

    2025

  4. [4]

    Hailing Cheng, Daqi Sun, and Xinyu Lu. 2026. Learning to Rotate: Tempo- ral and Semantic Rotary Encoding for Sequential Modeling.arXiv preprint arXiv:2604.24717(2026)

    Pith/arXiv arXiv 2026

  5. [5]

    Krzysztof Choromanski, Shanda Li, Valerii Likhosherstov, Kumar Avinava Dubey, Shengjie Luo, Di He, Yiming Yang, Tamás Sarlós, Thomas Weingarten, and Adrian Weller. 2024. Learning a Fourier Transform for Linear Relative Positional Encod- ings in Transformers. InInternational Conference on Artificial Intelligence and Sta- tistics, 2-4 May 2024, Palau de Con...

    2024

  6. [6]

    Krzysztof Choromanski, Han Lin, Haoxian Chen, Tianyi Zhang, Arijit Se- hanobish, Valerii Likhosherstov, Jack Parker-Holder, Tamás Sarlós, Adrian Weller, and Thomas Weingarten. 2022. From block-Toeplitz matrices to differential equa- tions on graphs: towards a general theory for scalable masked Transformers. InInternational Conference on Machine Learning, ...

    2022

  7. [7]

    Colwell, and Adrian Weller

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J. Colwell, and Adrian Weller. 2021. Rethinking Attention with Performers. In9th International Confer- ence on Learning Representations, ICLR 2021, Virt...

    2021

  8. [8]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv preprint arXiv:1810.04805(2018)

    Pith/arXiv arXiv 2018

  9. [9]

    Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. 2024. Rotary Position Embedding for Vision Transformer. InComputer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part X (Lecture Notes in Computer Science, Vol. 15068), Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattle...

    doi:10.1007/978-3-031-72684-2_17 2024

  10. [10]

    Gustav Herglotz. 1911. Über Potenzreihen mit positivem, reellem Teil im Ein- heitskreis.Berichte über die Verhandlungen der Königlich Sächsischen Gesellschaft der Wissenschaften zu Leipzig63 (1911), 501–511

    1911

  11. [11]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Jo- hannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556(2022)

    Pith/arXiv arXiv 2022

  12. [12]

    Bojian Hou, Xiaolong Liu, Xiaoyi Liu, Jiaqi Xu, Yasmine Badr, Mengyue Hang, Sudhanshu Chanpuriya, Junqing Zhou, Yuhang Yang, Han Xu, et al. 2026. Kunlun: Establishing scaling laws for massive-scale recommendation systems through unified architecture design.arXiv preprint arXiv:2602.10016(2026)

    Pith/arXiv arXiv 2026

  13. [13]

    Ermo Hua, Che Jiang, Xingtai Lv, Kaiyan Zhang, Youbang Sun, Yuchen Fan, Xuekai Zhu, Biqing Qi, Ning Ding, and Bowen Zhou. 2024. Fourier Position Embedding: Enhancing Attention’s Periodic Extension for Length Generalization. arXiv preprint arXiv:2412.17739(2024)

    Pith/arXiv arXiv 2024

  14. [14]

    Dai, Matthew D

    Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Cur- tis Hawthorne, Noam Shazeer, Andrew M. Dai, Matthew D. Hoffman, Monica Dinculescu, and Douglas Eck. 2019. Music Transformer: Generating Music with Long-Term Structure. In7th International Conference on Learning Repre- sentations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRevi...

    2019

  15. [15]

    Wang-Cheng Kang and Julian McAuley. 2018. Self-attentive sequential recom- mendation. In2018 IEEE international conference on data mining (ICDM). IEEE, 197–206

    2018

  16. [16]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361(2020)

    Pith/arXiv arXiv 2020

  17. [17]

    Seyed Mehran Kazemi, Rishab Goel, Sepehr Eghbali, Janahan Ramanan, Jaspreet Sahota, Sanjay Thakur, Stella Wu, Cathal Smyth, Pascal Poupart, and Marcus Brubaker. 2019. Time2vec: Learning a vector representation of time.arXiv preprint arXiv:1907.05321(2019)

    Pith/arXiv arXiv 2019

  18. [18]

    Byeongchan Kim, Arijit Sehanobish, Avinava Dubey, Min-hwan Oh, and Krzysztof Choromanski. 2026. RelFlexformer: Efficient Attention 3D- Transformers for Integrable Relative Positional Encodings.CoRRabs/2605.10706 (2026). arXiv:2605.10706 doi:10.48550/ARXIV.2605.10706

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.10706 2026

  19. [19]

    James Lee-Thorp, Joshua Ainslie, Ilya Eckstein, and Santiago Ontañón. 2021. FNet: Mixing Tokens with Fourier Transforms.arXiv preprint arXiv:2105.03824 (2021)

    Pith/arXiv arXiv 2021

  20. [20]

    Boyang Li, Yulin Wu, Sizhe Xu, Nuoxian Huang, Zhonghang Yuan, Shangyi Guo, Shu Yang, and Takahiro Yabe. 2026. nD-RoPE: A Generalized RoPE for n-Dimensional Position Embedding.arXiv preprint arXiv:2606.12146(2026)

    Pith/arXiv arXiv 2026

  21. [21]

    Jiacheng Li, Yujie Wang, and Julian McAuley. 2020. Time interval aware self- attention for sequential recommendation. InProceedings of the 13th international conference on web search and data mining. 322–330

    2020

  22. [22]

    Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontañón, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, and Srinadh Bho- janapalli. 2023. Functional Interpolation for Relative Positions Improves Long Context Transformers.CoRRabs/2310.04418 (2023). arXiv:2310.04418 doi:10.48550/ARXIV.2310.04418

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.04418 2023

  23. [23]

    Yang Li, Si Si, Gang Li, Cho-Jui Hsieh, and Samy Bengio. 2021. Learnable fourier features for multi-dimensional spatial positional encoding.Advances in Neural Information Processing Systems34 (2021), 15816–15829

    2021

  24. [24]

    Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, and Tie-Yan Liu. 2021. Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding. InAdvances in Neural Informa- tion Processing Systems 34: Annual Conference on Neural Information Process- ing Systems 2021, NeurIPS 2021, December 6-14, 20...

    2021

  25. [25]

    William Peebles and Saining Xie. 2023. Scalable diffusion models with transform- ers. InProceedings of the IEEE/CVF international conference on computer vision. 4195–4205

    2023

  26. [26]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.J. Mach. Learn. Res.21 (2020), 140:1–140:67. https://jmlr.org/papers/v21/20-074.html

    2020

  27. [27]

    Ali Rahimi and Benjamin Recht. 2007. Random features for large-scale kernel machines.Advances in neural information processing systems20 (2007)

    2007

  28. [28]

    Turner, and Petar Velickovic

    Isaac Reid, Arijit Sehanobish, Cederik Höfs, Bruno Mlodozeniec, Leonhard Vulpius, Federico Barbero, Adrian Weller, Krzysztof Choromanski, Richard E. Turner, and Petar Velickovic. 2026. Wavelet-Induced Rotary Encodings: RoPE Meets Graphs.ICMLabs/2509.22259 (2026). arXiv:2509.22259 doi:10.48550/ ARXIV.2509.22259

    Pith/arXiv arXiv doi:10.48550/arxiv.2509.22259 2026

  29. [29]

    Gritsenko, Matthias Minderer, Dmitry Kalashnikov, Jonathan Tompson, Vikas Sindhwani, and Krzysztof Marcin Choromanski

    Connor Schenck, Isaac Reid, Mithun George Jacob, Alex Bewley, Joshua Ainslie, David Rendleman, Deepali Jain, Mohit Sharma, Kumar Avinava Dubey, Ayzaan Wahid, Sumeet Singh, René Wagner, Tianli Ding, Chuyuan Fu, Arunkumar Byra- van, Jake Varley, Alexey A. Gritsenko, Matthias Minderer, Dmitry Kalashnikov, Jonathan Tompson, Vikas Sindhwani, and Krzysztof Marc...

    2025

  30. [30]

    Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. InProceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), Marilyn A. Walker, H...

    doi:10.18653/v1/n18-2074 2018

  31. [31]

    Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu

  32. [32]

    Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng

  33. [33]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. InAdvances in Neural Information Processing Systems (NeurIPS), Vol. 30

    2017

  34. [34]

    Xiaokai Wei, Jiajun Wu, Daiyao Yi, Reza Shirkavand, and Michelle Gong. 2025. Rotate Both Ways: Time-and-Order RoPE for Generative Recommendation.arXiv preprint arXiv:2510.20455(2025)

    arXiv 2025

  35. [35]

    Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan

  36. [36]

    Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, et al. 2024. Actions speak louder than words: Trillion-parameter sequential transducers for generative recommenda- tions.arXiv preprint arXiv:2402.17152(2024). A Proofs A.1 Proof of Proposition 3.2 Proof.Define the real-valued function: ℎ(𝜉 0,.....

    Pith/arXiv arXiv 2024

  37. [2019]

    Yiwen Chen, Joshua Ainslie, Krzysztof Choromanski, Xiang Gao, Su-Lin Wu, Yiping Yuan, and Qian Sun

    Self-attention with functional time representation learning.Advances in neural information processing systems32 (2019). Yiwen Chen, Joshua Ainslie, Krzysztof Choromanski, Xiang Gao, Su-Lin Wu, Yiping Yuan, and Qian Sun

    2019

  38. [2020]

    Fourier features let networks learn high frequency functions in low di- mensional domains.Advances in neural information processing systems33 (2020), 7537–7547

    2020

  39. [2021]

    RoFormer: Enhanced Transformer with Rotary Position Embedding.arXiv preprint arXiv:2104.09864(2021)

    Pith/arXiv arXiv 2021