Pith. sign in

REVIEW 13 cited by

An Attention Free Transformer

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2105.14103 v2 pith:LMRPHLBC submitted 2021-05-28 cs.LG cs.CLcs.CV

Shuangfei Zhai , Walter Talbott , Nitish Srivastava , Chen Huang , Hanlin Goh , Ruixiang Zhang , Josh Susskind This is my paper · ORCID
classification cs.LGcs.CLcs.CV
keywords attentionfreeintroducemodeltransformerwhileadvantageaft-conv
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

We introduce Attention Free Transformer (AFT), an efficient variant of Transformers that eliminates the need for dot product self attention. In an AFT layer, the key and value are first combined with a set of learned position biases, the result of which is multiplied with the query in an element-wise fashion. This new operation has a memory complexity linear w.r.t. both the context size and the dimension of features, making it compatible to both large input and model sizes. We also introduce AFT-local and AFT-conv, two model variants that take advantage of the idea of locality and spatial weight sharing while maintaining global connectivity. We conduct extensive experiments on two autoregressive modeling tasks (CIFAR10 and Enwik8) as well as an image recognition task (ImageNet-1K classification). We show that AFT demonstrates competitive performance on all the benchmarks, while providing excellent efficiency at the same time.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 13 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. NAE: Normalizing AutoEncoder

    cs.LG 2026-08 conditional novelty 6.0 of 10

    A conditional surrogate loss that always picks the gradient estimate aligned with the reconstruction loss improves flow autoencoder training and reaches state-of-the-art generative performance on molecules, tabular da...

  2. Desc++: Efficient Descriptor Enhancement for Data Association in Existing Visual SLAM Systems

    cs.RO 2026-07 accept novelty 6.0 of 10

    A hybrid Mamba–attention-free transformer module enhances handcrafted and learned descriptors in linear time, improving data association and trajectory accuracy when plugged into four existing V-SLAM systems.

  3. Multi-View Learning with Context-Guided Receptance for Image Denoising

    eess.IV 2025-05 conditional novelty 6.0 of 10

    CRWKV, a bidirectional RWKV network with context-guided token shift and frequency mixing, matches or modestly exceeds prior real-image denoisers with lower computational cost.

  4. MetaLA: Unified Optimal Linear Approximation to Softmax Attention Map

    cs.LG 2024-11 reject novelty 6.0 of 10

    MetaLA removes the Key matrix from linear attention, adds input-dependent decay plus self-augmentation, and reports gains over many linear baselines.

  5. MixFormer: Linear Transformer with Mixture of Memory Experts

    cs.LG 2026-08 reject novelty 5.0 of 10

    A new linear attention architecture with memory experts is described, but its equations are internally inconsistent and the reported results lack the detail needed to verify them.

  6. Med-URWKV{\dag}: Toward Enhanced Pretrained Pure VRWKV Models for Medical Image Segmentation

    eess.IV 2025-06 conditional novelty 5.0 of 10

    Pretrained pure VRWKV encoders paired with pure VRWKV decoders match or beat CNN, ViT, and Mamba baselines, with a small model plus FAWA and MSCF modules reaching 88% average Dice.

  7. Attention-based Neural Network Emulators for Multi-Probe Data Vectors Part III: Modeling The Next Generation Surveys

    astro-ph.CO 2025-05 conditional novelty 5.0 of 10

    A transformer-based emulator reproduces CAMB CMB TT, TE, and EE power spectra within cosmic variance errors across a wide Lambda-CDM parameter space, with outlier fractions below 10% for future survey configurations.

  8. FE-MCFormer: a novel time-frequency interpretable architecture for machinery fault diagnosis under strong noise environments

    eess.SP 2025-05 conditional novelty 5.0 of 10

    FE-MCFormer, a time-frequency transformer-style network with a learnable Fourier reconstruction layer, achieves state-of-the-art fault diagnosis accuracy on three rotating machinery datasets under strong noise.

  9. ReGLA: Refining Gated Linear Attention

    cs.CL 2025-02 conditional novelty 5.0 of 10

    ReGLA combines a bounded exponential feature map, a log-normal variance scaling factor, and a refined forget gate to improve gated linear attention perplexity on WikiText-103 and post-linearized Pythia-160M.

  10. Hadamard product in deep learning: Introduction, Advances and Challenges

    cs.LG 2025-04 conditional novelty 4.0 of 10

    A survey that taxonomizes deep learning applications of the Hadamard product into four domains and argues it is a core architectural primitive.

  11. An Efficient Attention Mechanism for Sequential Recommendation Tasks: HydraRec

    cs.IR 2025-01 conditional novelty 4.0 of 10

    HydraRec applies the existing Hydra attention mechanism to the BERT4Rec sequential recommender, reporting faster training and competitive or better accuracy on three datasets.

  12. Efficient Relational Context Perception for Knowledge Graph Completion

    cs.LG 2024-12 conditional novelty 3.0 of 10

    A knowledge graph completion model that processes each triple through RWKV-style mixing layers and a Tucker decoder reports state-of-the-art or near-state-of-the-art scores on FB15k, YAGO3-10, UMLS, and FB13.

  13. A Survey of RWKV

    cs.CL 2024-12 conditional novelty 3.0 of 10

    A review of the RWKV architecture, its versions, applications, benchmarks, and open-source ecosystem; it presents no new experimental results.

Pith tools