Pith. sign in

REVIEW 3 cited by

EcoTransformer: Attention without Multiplication

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2507.20096 v2 pith:S7WKVTIS submitted 2025-07-27 cs.LG cs.AIcs.CL

Xin Gao , Xingming Xu , Shirin Amiraslani , Hong Xu This is my paper
classification cs.LGcs.AIcs.CL
keywords attentiondot-productarchitectureecotransformerenergymechanismmultiplicationscaled
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The Transformer, with its scaled dot-product attention mechanism, has become a foundational architecture in modern AI. However, this mechanism is computationally intensive and incurs substantial energy costs. We propose a new Transformer architecture EcoTransformer, in which the output context vector is constructed as the convolution of the values using a Laplacian kernel, where the distances are measured by the L1 metric between the queries and keys. Compared to dot-product based attention, the new attention score calculation is free of matrix multiplication. It performs on par with, or even surpasses, scaled dot-product attention in NLP, bioinformatics, and vision tasks, while consuming significantly less energy. (This version (v2) supersedes v1 and reflects the intended release and licensing.)

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

    cs.CL 2026-06 unverdicted novelty 6.0 of 10

    Keyless Attention eliminates the key projection in transformers via value-space routing, cutting KV cache size by 50% with competitive or superior performance on language modeling and zero-shot benchmarks.

  2. Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

    cs.CL 2026-06 conditional novelty 5.0 of 10

    Keyless Attention eliminates the key projection, computing softmax(QV^T)V and reducing KV-cache memory by exactly 50%, with comparable or better performance on small-scale models.

  3. Keyless Attention: Value-Space Routing and Value-Only Caching for Efficient Transformers

    cs.CL 2026-06 conditional novelty 4.0 of 10

    An attention variant that uses values as keys reduces KV-cache memory by 50% and shows roughly comparable perplexity on small-scale training runs.

Pith tools