Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

ZETA: Leveraging Z-order Curves for Efficient Top-k Attention

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that top-k attention can be made parallel across entire sequences under causal masks by sorting Z-order projections of low-dimensional keys and queries, and that the resulting method matches or outperforms standard…

desk verdict The Z-order idea is fresh and the experiments are real, but the causal-masking procedure as written is self-contradictory and the main theoretical bound goes imaginary in exactly the regime the paper uses, so the central claims don't hold up. read the letter →

arxiv 2501.14577 v3 pith:H2A5K2HQ submitted 2025-01-24 cs.LG cs.AI

Qiuhao Zeng , Jerry Huang , Peng Lu , Gezheng Xu , Boxing Chen , Charles Ling , Boyu Wang This is my paper · ORCID
classification cs.LGcs.AI
keywords top-kattentionZ-ordercurveMortoncodecausalmaskingsparselong-sequencetransformersCauchykernelnear-neighborsearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ZETA sets out to remove a bottleneck in top-$k$ attention: with a causal mask, every query may only attend to past tokens, and earlier methods either mask after selection (leaving early queries with nothing to attend to) or walk through the sequence token by token, losing parallelism. The paper's solution is to project keys and queries into a low-dimensional space, map them to one-dimensional Z-order codes, sort the codes once, and let each query binary-search for its own code and read a fixed window of neighbours from the sorted prefix. If this works, training and inference for sparse attention run in $O(N\log N)$ time and memory while keeping the whole sequence parallel on accelerators. The paper backs this with a trade-off analysis for key/query dimensionality, a trainable Cauchy softmax over Euclidean distances, and empirical results where ZETA matches vanilla attention on associative recall, posts the best average Long Range Arena accuracy among the compared models, and reaches near-Transformer perplexity on WikiText-103.

What carries the argument

The load-bearing object is the Z-order curve (Morton code), which maps a $d$-dimensional point to a one-dimensional integer by interleaving the bits of its coordinates so that nearby points tend to get nearby codes. ZETA sorts the key codes once, splits the sorted list into chunks, and for each query in chunk $m$ binary-searches the query's code inside the first $m$ chunks and reads a fixed-size window around that insertion position as the top-$k$ set. The Adaptive Cauchy-Softmax then converts Euclidean distances in the low-dimensional space into attention weights through $\frac{1}{\|q-k\|^2 + \gamma^2}$, with $\gamma$ trainable per layer.

What would settle it

Run ZETA on a trained checkpoint, compute each query's exact $k$ nearest keys by Euclidean distance on the same low-dimensional keys, and measure the recall of ZETA's selected indices against that exact set; if recall is low on real data, the windowed-contiguity assumption fails.

Watch

Extended reading notes

Core claim

On its own terms, ZETA's central claim is that the top-$k$ attended tokens for every query in a causally masked sequence can be found in parallel by sorting all keys once in Z-order space, splitting the sorted list into chunks, and restricting each query in chunk $m$ to keys in the first $m$ chunks; the query's insertion position in that prefix locates a window of $k$ nearest neighbours. To make this geometry meaningful, the paper argues that keys and queries should have much lower dimension than values: small $d_K$ preserves relative distances after Z-order projection (supported by a bound built on the Johnson–Lindenstrauss lemma and a Lipschitz regression risk analysis), while a large $d_V$ preserves semantic richness. Attention weights are then computed with an Adaptive Cauchy-Softmax over Euclidean distances, with a trainable $\gamma$ controlling the receptive field, and the paper reports that ZETA matches vanilla attention on Multi-Query Associative Recall, achieves the best average accuracy among the compared models on Long Range Arena, reaches 26.3 perplexity on WikiText-103 against 26.2 for vanilla attention, and runs faster than a FlashAttention baseline at long sequence lengths.

Load-bearing premise

The central assumption is that the true top-$k$ keys for a query are contiguous around the query's insertion position in the sorted Z-order list, so a fixed window over the first $m$ chunks recovers them without scanning the full prefix.

Editorial extensions

If this is right

  • A causal top-$k$ attention layer can be trained in parallel over the full sequence, with sorting dominating the cost at $O(N\log N)$ time and space rather than the $O(N^2)$ of standard attention.
  • Key and query dimensions can be set far below value dimensions ($d_K = d_Q = 3$ in the paper's experiments) without degrading accuracy, because matching needs relative-distance preservation rather than semantic richness.
  • Replacing the exponential softmax with a trainable Cauchy kernel over Euclidean distances keeps long-range tokens influential while letting each layer adjust its receptive field.
  • On the reported benchmarks ZETA matches vanilla attention on Multi-Query Associative Recall, posts the best average LRA accuracy among the compared models, and reaches 26.3 WikiText-103 perplexity versus 26.2 for vanilla attention.
  • At sequence lengths from 8K to 64K, the paper's custom-kernel implementation is faster than FlashAttention in forward and forward-backward passes, with modestly higher memory use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, a natural stress test is to measure recall@k of ZETA's window retrieval against exact kNN on real trained keys; if recall is low, a data-dependent window size or a candidate re-ranking stage would be a direct fix.
  • The chunk-prefix causal rule may discard past keys whose Z-order codes land in later chunks; an alternative is to sort each prefix independently or use a hierarchical index, trading speed for exactness.
  • If the low-dimensional matching insight generalizes, other distance-based sparse attention schemes (LSH, product quantization, locality-sensitive retrieval) could adopt the same split between small matching dimensions and large value dimensions.
  • Because the Cauchy kernel's $\gamma$ is trained per layer, one could test whether learned receptive fields correlate with task structure: sharp in local tasks, broad in long-range retrieval tasks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes ZETA, a sparse top-k attention mechanism that maps keys and queries to a low-dimensional space, then to one dimension via Z-order curves, sorts the keys, and partitions them into chunks to enable parallel top-k retrieval under causal masks. It claims O(N log N) time and space complexity and reports experiments on the Multi-Query Associative Recall task, Long Range Arena, and WikiText-103, along with a theoretical analysis motivating small key/query dimensionality and an adaptive Cauchy softmax. The paper is written as an ICLR 2025 conference paper and includes a custom Triton implementation with efficiency measurements.

Significance. If the algorithm worked as described, ZETA would address a genuine bottleneck: causal top-k attention is difficult to parallelize because future tokens must be excluded before retrieval. The paper also contains a substantial empirical component and a custom Triton implementation with reported speedups. However, the central algorithmic mechanism is internally inconsistent (see Major Comment 1), and the theoretical bound used to justify the low-dimensional projection is invalid in the relevant regime (Major Comment 2). These issues undermine the paper's main claims, so the current manuscript cannot be accepted.

major comments (3)
  1. [Section 3.2.2 and Algorithm 1] The described causal chunked search is internally inconsistent. After globally sorting the Z-order keys, the sorted list is a permutation of all N original positions. The paper says that a query in chunk m searches only the first m chunks and 'index[es] the original unsorted keys from 0 to m×M−1 in the sorted list,' and Algorithm 1 Step 4 says to 'Exclude keys from positions j > m×M.' These two conditions cannot hold simultaneously: the first m chunks of the sorted order will generally contain keys with original indices larger than m×M−1, and after excluding them, the number of valid keys is typically much less than k and often zero for early queries. Consequently, the method as specified does not retrieve k past keys per query and does not realize the claimed O(N log N) parallel causal top-k attention. Sorting each prefix separately would cost O(N^2 log N), and no incremental data structure is specified; thus the central efficiency claim is unsupported.
  2. [Section 3.2.1 and Appendix A] The risk bound in Theorem 3.3 contains the factor sqrt(1 − sqrt(C ln m / dK)). In the proof, epsilon is set to sqrt(C ln m / dK) in the Johnson–Lindenstrauss Lemma, which is only valid for epsilon < 1. For the paper's recommended dK = 3, this requires m < exp(3/C); for the sequence lengths considered in the paper (e.g., N = 4096 in LRA), m will typically exceed this threshold, making the bound imaginary and the derivation invalid. The theorem therefore does not establish the claimed trade-off between the curse of dimensionality and preservation of relative distances in the regime where ZETA operates.
  3. [Section 3.2.2] The nearest-neighbor retrieval step selects a window of size k around the query's insertion position in the sorted Z-order list and treats the keys in that window as the top-k set. The paper supplies no bound or proof that the Euclidean top-k neighbors are contained in such a window after sorting; Z-order curves preserve locality only approximately, and the locality experiment in Section 4.4 measures neighbor overlap after projection, not contiguity after sorting. Without a retrieval-recall guarantee, the method is not demonstrably a top-k attention method, and the experimental accuracy cannot be attributed to the described selection mechanism.
minor comments (5)
  1. [Appendix A] The paragraph before Lemma A.1 contains an incomplete sentence and an unresolved URL: 'The recent paper, Reformer, proposed the Shared-QK Transformer... https://arxiv.org/abs/2001.0445a very simple but efficient technique.' This appears to be a leftover note and should be rewritten or removed.
  2. [Appendix D] There is a typo in 'he @triton.autotune decorator is used...' which should read 'The @triton.autotune decorator is used...'.
  3. [Table 2] The claim of consistent improvement over baselines is qualified by the Pathfinder row: ZETA's accuracy (68.20) is lower than that of the vanilla Transformer (71.40) and several other variants. The narrative in Section 4.1 should be adjusted to reflect this.
  4. [Section 3.2.2] The notation is confusing: the top-k value is sometimes written as k and sometimes as K (e.g., in Section 3.2.2, 'a window of size K centered around the insertion point'). Please standardize the notation.
  5. [Appendix F] The Limitations section acknowledges the general risk of ignoring important tokens in top-k attention, but it does not mention the causal-masking inconsistency described in Major Comment 1, which is a more direct limitation of the proposed algorithm.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ZETA's contributions are an algorithmic construction plus empirical evaluation; the theoretical bound is derived from the external Johnson–Lindenstrauss lemma and standard Lipschitz/simplex assumptions, and its tunable quantities are hyperparameters, not fitted constants relabeled as predictions.

full rationale

The paper's claimed derivation chain is self-contained rather than circular. ZETA's central mechanism is algorithmic: keys are projected to one dimension via Z-order curves, sorted, chunked, and searched by insertion position (Section 3.2.2 and Algorithm 1). No prediction in the paper is obtained by fitting a parameter to a target quantity and then reporting that quantity; dK is chosen by ablations (Section 4.4 and Appendix G.1), k is set to 32 after an ablation (Section 4.5), and the Cauchy softmax scale gamma is a trainable parameter (Section 3.3 and Appendix E). The theoretical claim about dK (Theorem 3.3) is derived from the Johnson–Lindenstrauss lemma, Lipschitzness, and a covering argument with an external textbook lemma (Lemma A.4), not from a result whose authors overlap with this paper. The only self-citations appear in the introduction as routine prior-work mentions (Zeng et al. 2023; 2024a; 2024b; Fang et al. 2025) and are not load-bearing. The causal-masking inconsistency raised by a skeptical reading (a single global sort cannot simultaneously supply the first-m-chunk prefix and the original-position exclusion j > m*M) is a substantive correctness concern, but it is not a circularity: the algorithm does not define its output in terms of the quantity it claims to predict. Similarly, the possible invalidity of the sqrt(1 - sqrt(C ln m / dK)) bound at dK = 3 is a mathematical gap, not a circular reduction. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

ZETA rests on several unproved domain assumptions most importantly that Z-order locality makes window-based top-k retrieval exact enough, and that causal masking can be enforced with a single global sort. The free parameters dK, k, gamma, and chunk count are chosen empirically. The intended-entity ledger is empty because the paper introduces no new physical or conceptual entity, only a new attention kernel and an algorithm.

free parameters (4)
  • key/query dimension dK = 3
    Chosen empirically from locality-preservation plots (Figure 3) and LRA ablations (Table 5). The theory is not valid at this value for long contexts, so the choice is effectively fitted.
  • top-k window size k = 32
    Default value in most experiments; Figure 2d shows little sensitivity across k in [16, 48], so it is a hand-set hyperparameter.
  • Cauchy kernel width gamma = trainable, sigmoid output in [0, 1]
    Learned per layer to control the receptive field of the Cauchy softmax. It is a model parameter, not derived from first principles.
  • number of chunks = 4, 8, 16, 32
    Set per sequence length in Appendix C as a configuration choice; no principled derivation is given.
assumptions (6)
  • standard math Johnson-Lindenstrauss lemma with dK = Omega(ln m / epsilon^2)
    Invoked in Section 3.2.1 and Appendix A to justify low-dimensional key and query projections. The lemma requires dK to grow with log m, which conflicts with the recommended dK=3 for long sequences.
  • ad hoc to paper Assumption 3.2: an optimal learnable similarity critic Gamma exists and attention weights lie in a simplex
    Assumed to derive the risk bound. This is a strong, unproved modeling assumption about the attention hypothesis class.
  • domain assumption Z-order curves preserve locality well enough that a fixed-size window around the insertion position contains the true top-k nearest neighbors
    Central to the retrieval step in Section 3.2.2. No theorem or precise bound is provided; only empirical overlap measurements in Figure 3.
  • ad hoc to paper The chunked causal search can be implemented with a single global sort while excluding future original positions
    Algorithm 1 and Section 3.2.2 describe taking the first m chunks of a globally sorted list while also excluding original positions j > m*M. These are inconsistent for a global permutation, and per-prefix sorting would change the complexity.
  • domain assumption Euclidean distance with a Cauchy kernel can replace dot-product softmax attention
    The method replaces softmax with a trainable Cauchy kernel in Section 3.3. The equivalence to standard attention is not proven; the paper argues by example and experiments.
  • domain assumption The conditional mean function h* is l-Lipschitz and projections are bounded by B
    These are standard learning-theory assumptions used in Theorem 3.3 and Appendix A, but they are not verified for the real data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ZETA: Leveraging Z-order Curves for Efficient Top-k Attention." pith.science (2026). https://pith.science/paper/H2A5K2HQ

@misc{pith2026250114577,
  author       = {Pith},
  title        = {Pith review of: ZETA: Leveraging Z-order Curves for Efficient Top-k Attention},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/H2A5K2HQ}},
  note         = {Machine review of arXiv:2501.14577}
}
abstract

Over recent years, the Transformer has become a fundamental building block for sequence modeling architectures. Yet at its core is the use of self-attention, whose memory and computational cost grow quadratically with the sequence length $N$, rendering it prohibitively expensive for long sequences. A promising approach is top-$k$ attention, which selects only the $k$ most relevant tokens and achieves performance comparable to vanilla self-attention while significantly reducing space and computational demands. However, causal masks require the current query token to only attend to past tokens, preventing the existing top-$k$ attention method from efficiently searching for the most relevant tokens in parallel, thereby limiting training efficiency. In this work, we propose ZETA, leveraging \textbf{Z}-Order Curves for \textbf{E}fficient \textbf{T}op-$k$ \textbf{A}ttention, to enable parallel querying of past tokens for entire sequences. % in both space and time complexity of $\mathcal{O}(N \log N)$. We first theoretically show that the choice of key and query dimensions involves a trade-off between the curse of dimensionality and the preservation of relative distances after projection. In light of this insight, we propose reducing the dimensionality of keys and queries in contrast to values and further leverage $Z$-order curves to map low-dimensional keys and queries into \emph{one}-dimensional space, which permits parallel sorting, thereby largely improving the efficiency for top-$k$ token selection. Experimental results demonstrate that ZETA matches the performance of standard attention on the synthetic \textsc{Multi-Query Associative Recall} task and outperforms attention and its variants on \textsc{Long Range Arena} and \textsc{WikiText-103} language modeling.

Figures

Figures reproduced from arXiv: 2501.14577 by the authors.

Figure 1
Figure 1. Illustration of attention using Eu￾clidean distance vs. dot product. Eu￾clidean distance correctly classifies points into classes ±1, while the dot product leads to a misclassified area. misclassified area will be classified as “+1” using the dot-product metric). Second, k-NN search is typ￾ically based on the Euclidean metric, while using the dot-product requires normalization that loses token magnitudes. To better … view at source ↗
Figure 2
Figure 2. Experiments on Associative Recall: (a) Model Accuracy (b) Performance of Transformer [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. The effect of dimensional￾ity reduction before and after Z-order curves projection on locality preserva￾tion for different sample sizes. Next, we evaluate how well Z-order curve projections preserve locality across different dimensions and sam￾ple sizes. Specifically, we test the locality preserva￾tion by measuring the overlap between the top-64 nearest neighbors before and after projection, with sample sizes N ∈ {5… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Illustration of the chunking process in ZETA: Keys are projected into one-dimensional [PITH_FULL_IMAGE:figures/full_fig_p019_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks

    cs.CL 2025-05 conditional novelty 6.0 of 10

    FlowHN splits input tokens between parallel attention and SSM branches to balance compute load, improving throughput and hardware utilization in small autoregressive language models.

Reference graph

Works this paper leans on

59 extracted references · 31 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Principal component analysis

    Herv \'e Abdi and Lynne J Williams. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2 0 (4): 0 433--459, 2010

    work page 2010

  3. [3]

    Zoology: Measuring and improving recall in efficient language models

    Simran Arora, Sabri Eyuboglu, Aman Timalsina, Isys Johnson, Michael Poli, James Zou, Atri Rudra, and Christopher Re. Zoology: Measuring and improving recall in efficient language models. In The Twelfth International Conference on Learning Representations, 2024 a . URL https://openreview.net/forum?id=LY3ukUANko

    work page 2024

  4. [4]

    Simple linear attention language models balance the recall-throughput tradeoff

    Simran Arora, Sabri Eyuboglu, Michael Zhang, Aman Timalsina, Silas Alberti, James Zou, Atri Rudra, and Christopher Re. Simple linear attention language models balance the recall-throughput tradeoff. In Forty-first International Conference on Machine Learning, 2024 b

    work page 2024

  5. [5]

    Neural machine translation by jointly learning to align and translate

    Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations, 2015

    work page 2015

  6. [6]

    Longformer: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  7. [7]

    Amanda Bertsch, Uri Alon, Graham Neubig, and Matthew R. Gormley. Unlimiformer: Long-range transformers with unlimited length input. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=lJWUJWLCJo

    work page 2023

  8. [8]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pp.\ 2397--2430. PMLR, 2023

    2023

Show all 59 references
  1. [9]

    Probability and Measure

    Patrick Billingsley. Probability and Measure. John Wiley and Sons, second edition, 1986

    1986

  2. [10]

    Video generation models as world simulators, 2024

    Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufei Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng, Ricky Wang, and Aditya Ramesh. Video generation models as world simulators, 2024. URL https://openai.com/research/video-generation-models-...

    2024

  3. [11]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffr...

    2020

  4. [12]

    Skyformer: Remodel self-attention with gaussian kernel and nystr \"o m method

    Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. Skyformer: Remodel self-attention with gaussian kernel and nystr \"o m method. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.ne...

    2021

  5. [13]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  6. [14]

    Rethinking attention with performers

    Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, David Benjamin Belanger, Lucy J Colwell, and Adrian Weller. Rethinking attention with performers. In I...

    2021

  7. [15]

    Cover and Joy A

    Thomas M. Cover and Joy A. Thomas. Elements of information theory. John Wiley & Sons, 2006

    2006

  8. [16]

    Flash A ttention-2: Faster attention with better parallelism and work partitioning

    Tri Dao. Flash A ttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR), 2024

    2024

  9. [17]

    Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e

    Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher R \'e . Flash A ttention: Fast and memory-efficient exact attention with IO -awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  10. [18]

    BERT : Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT : Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North A merican Chapter of...

    2019 doi

  11. [19]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  12. [20]

    Topology

    James Dugundji. Topology. Allyn and Bacon, 1966

    1966

  13. [21]

    Structure-preserving graph representation learning

    Ruiyi Fang, Liangjian Wen, Zhao Kang, and Jianzhuang Liu. Structure-preserving graph representation learning. In 2022 IEEE International Conference on Data Mining (ICDM), pp.\ 927--932. IEEE, 2022

    2022

  14. [22]

    On the benefits of attribute-driven graph domain adaptation

    Ruiyi Fang, Bingheng Li, Zhao Kang, Qiuhao Zeng, Ruizhi Pu, Nima Hosseini Dashtbayaz, Boyu Wang, and Charles Ling. On the benefits of attribute-driven graph domain adaptation. The Thirteenth International Conference on Learning Representations, 2025

    2025

  15. [23]

    Mamba: Linear-time sequence modeling with selective state spaces

    Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. In First Conference on Language Modeling, 2024. URL https://openreview.net/forum?id=tEYskw1VY2

    2024

  16. [24]

    Memory-efficient transformers via top-k attention

    Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. Memory-efficient transformers via top-k attention. In Nafise Sadat Moosavi, Iryna Gurevych, Angela Fan, Thomas Wolf, Yufang Hou, Ana Marasovi \'c , and Sujith Ravi (eds.), Proceedings of the Second Worksho...

    2021 doi

  17. [25]

    Axial attention in multidimensional transformers, 2020

    Jonathan Ho, Nal Kalchbrenner, Dirk Weissenborn, and Tim Salimans. Axial attention in multidimensional transformers, 2020. URL https://openreview.net/forum?id=H1e5GJBtDr

    2020

  18. [26]

    Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, L \' e lio Renard Lavaud, Lucile Saulnier, Marie...

    2024 doi arXiv

  19. [27]

    Extensions of lipschitz maps into banach spaces

    William B Johnson, Joram Lindenstrauss, and Gideon Schechtman. Extensions of lipschitz maps into banach spaces. Israel Journal of Mathematics, 54 0 (2): 0 129--138, 1986

    1986

  20. [28]

    Daniel Jurafsky and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models. Pearson, 3rd edition, 2024. URL https://web.stanford.edu/ jurafsky/slp3/

    2024

  21. [29]

    Reformer: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=rkgNKkHtvB

    2020

  22. [30]

    Iceformer: Accelerated inference with long-sequence transformers on CPU s

    Yuzhen Mao, Martin Ester, and Ke Li. Iceformer: Accelerated inference with long-sequence transformers on CPU s. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=6RR3wU4mSZ

    2024

  23. [31]

    Umap: Uniform manifold approximation and projection

    Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3 0 (29): 0 861, 2018

    2018

  24. [32]

    Pointer sentinel mixture models

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Byj72udxe

    2017

  25. [33]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

    2024 arXiv

  26. [34]

    Scaling neural machine translation

    Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Ond r ej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, A...

    2018

  27. [35]

    Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas K \" o pf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Stei...

    2019

  28. [36]

    The devil in linear transformer

    Zhen Qin, Xiaodong Han, Weixuan Sun, Dongxu Li, Lingpeng Kong, Nick Barnes, and Yiran Zhong. The devil in linear transformer. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNL...

    2022 doi

  29. [37]

    cosformer: Rethinking softmax in attention

    Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong. cosformer: Rethinking softmax in attention. In International Conference on Learning Representations, 2022 b . URL https://openreview.net/forum?id=Bl8CQrx2Up4

    2022

  30. [38]

    Language models are unsupervised multitask learners, 2019

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners, 2019

    2019

  31. [39]

    Zero-shot text-to-image generation

    Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 ...

    2021

  32. [40]

    Understanding Machine Learning: From Theory to Algorithms

    Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms. Cambridge University Press, 2014

    2014

  33. [41]

    A study on relu and softmax in transformer

    Kai Shen, Junliang Guo, Xu Tan, Siliang Tang, Rui Wang, and Jiang Bian. A study on relu and softmax in transformer. CoRR, abs/2302.06461, 2023. doi:10.48550/ARXIV.2302.06461. URL https://doi.org/10.48550/arXiv.2302.06461

    2023 doi arXiv

  34. [42]

    Sparse S inkhorn attention

    Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. Sparse S inkhorn attention. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp.\ 9438--9447. ...

    2020

  35. [43]

    Long range arena : A benchmark for efficient transformers

    Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference on Learning Representations, 2021. URL https://openr...

    2021

  36. [44]

    Efficient transformers: A survey

    Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. ACM Comput. Surv., 55 0 (6), December 2022. ISSN 0360-0300. doi:10.1145/3530811. URL https://doi.org/10.1145/3530811

    2022 doi

  37. [45]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers & distillation through attention. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machin...

    2021

  38. [46]

    Gomez, ukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Inf...

    2017

  39. [47]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  40. [48]

    o mformer: A nystr \

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nystr \"o mformer: A nystr \"o m-based algorithm for approximating self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021

    2021

  41. [49]

    Big bird: Transformers for longer sequences

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. Big bird: Transformers for longer sequences. 33: 0 17283--17297, 2020. URL https://proceedings.neurips.cc/pape...

    2020

  42. [50]

    Foresee what you will learn: data augmentation for domain generalization in non-stationary environment

    Qiuhao Zeng, Wei Wang, Fan Zhou, Charles Ling, and Boyu Wang. Foresee what you will learn: data augmentation for domain generalization in non-stationary environment. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp.\ 11147--11155, 2023

    2023

  43. [51]

    Towards understanding evolving patterns in sequential data

    QIUHAO Zeng, Long-Kai Huang, Qi Chen, Charles X Ling, and Boyu Wang. Towards understanding evolving patterns in sequential data. Advances in Neural Information Processing Systems, 37: 0 132747--132773, 2024 a

    2024

  44. [52]

    Latent trajectory learning for limited timestamps under distribution shift over time

    QIUHAO Zeng, Changjian Shui, Long-Kai Huang, Peng Liu, Xi Chen, Charles Ling, and Boyu Wang. Latent trajectory learning for limited timestamps under distribution shift over time. In The Twelfth International Conference on Learning Representations, 2024 b . URL https://openrevi...

    2024

  45. [53]

    Susskind

    Shuangfei Zhai, Tatiana Likhomanenko, Etai Littwin, Dan Busbridge, Jason Ramapuram, Yizhe Zhang, Jiatao Gu, and Joshua M. Susskind. Stabilizing transformer training by preventing attention entropy collapse. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, ...

    2023

  46. [54]

    The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry

    Michael Zhang, Kush Bhatia, Hermann Kumbong, and Christopher Re. The hedgehog & the porcupine: Expressive linear attentions with softmax mimicry. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=4g02l2N2Nx

    2024

  47. [55]

    H -transformer-1 D : Fast one-dimensional hierarchical attention for sequences

    Zhenhai Zhu and Radu Soricut. H -transformer-1 D : Fast one-dimensional hierarchical attention for sequences. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th...

    2021 doi

  48. [56]

    Efficient attention: Attention with linear complexities

    Shen Zhuoran, Zhang Mingyuan, Zhao Haiyu, Yi Shuai, and Li Hongsheng. Efficient attention: Attention with linear complexities. In 2021 IEEE Winter Conference on Applications of Computer Vision (WACV), 2021

    2021

  49. [57]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  50. [58]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  51. [59]

    = Y S RF8yV:Ĉ` LB , JDP P[; QX,PueK6ڞ G:EZt x( ɠOU(T P w 3Ѝ * SKE <y فB0;a \!hi BC=lS;6m9h'VlSh D:E L ܓo K i N )pI[ wbk

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.