REVIEW 8 cited by
Flowformer: Linearizing Transformers with Conservation Flows
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Transformers based on the attention mechanism have achieved impressive success in various areas. However, the attention mechanism has a quadratic complexity, significantly impeding Transformers from dealing with numerous tokens and scaling up to bigger models. Previous methods mainly utilize the similarity decomposition and the associativity of matrix multiplication to devise linear-time attention mechanisms. They avoid degeneration of attention to a trivial distribution by reintroducing inductive biases such as the locality, thereby at the expense of model generality and expressiveness. In this paper, we linearize Transformers free from specific inductive biases based on the flow network theory. We cast attention as the information flow aggregated from the sources (values) to the sinks (results) through the learned flow capacities (attentions). Within this framework, we apply the property of flow conservation into attention and propose the Flow-Attention mechanism of linear complexity. By respectively conserving the incoming flow of sinks for source competition and the outgoing flow of sources for sink allocation, Flow-Attention inherently generates informative attentions without using specific inductive biases. Empowered by the Flow-Attention, Flowformer yields strong performance in linear time for wide areas, including long sequence, time series, vision, natural language, and reinforcement learning. The code and settings are available at this repository: https://github.com/thuml/Flowformer.
Forward citations
Cited by 8 Pith papers
-
FreSH: Frequency-Segmented Hierarchical Multi-Expert Framework for Multivariate Time Series Classification
FreSH, a frequency-segmented multi-expert network, reports the highest average accuracy (76.1%) across 30 UEA multivariate time series classification benchmarks with a compact 54k-parameter model.
-
FreEformer: Frequency Enhanced Transformer for Multivariate Time Series Forecasting
A frequency-domain Transformer with an augmented attention matrix beats or matches state-of-the-art forecasters on many time series benchmarks, though the SOTA claim is not consistent across all datasets.
-
LLT: Local Linear Transformer for PDE Operator Learning
Local Linear Transformer learns PDE operators by combining linear global attention with local spatial mixing, achieving competitive accuracy and lower training-step cost than prior transformers.
-
DIFFUMA: High-Fidelity Spatio-Temporal Video Prediction via Dual-Path Mamba and Diffusion Enhancement
DIFFUMA couples a bidirectional Mamba temporal backbone with a single-pass diffusion-style denoiser on a new chip dicing lane dataset, reporting large MSE and SSIM gains over prior video predictors.
-
Learning Advanced Self-Attention for Linear Transformers in the Singular Value Domain
AGF replaces softmax attention with a learned singular-value-domain graph filter that runs in O(n d^2), and reports moderate accuracy improvements on UEA and LRA benchmarks.
-
SCFormer: Structured Channel-wise Transformer with Cumulative Historical State for Multivariate Time Series Forecasting
SCFormer, a channel-wise Transformer with triangular or convolutional temporal constraints and a HiPPO cumulative history state, improves forecasting on several benchmarks but not on Traffic.
-
Gateformer: Advancing Multivariate Time Series Forecasting through Temporal and Variate-Wise Attention with Gated Representations
Gateformer combines temporal patching attention, variate-wise attention, and two gating mechanisms to achieve the best average forecasting error on most of 13 multivariate benchmarks, and it reports improved accuracy ...
-
PaMMA-Net: Plasmas magnetic measurement evolution based on data-driven incremental accumulative prediction
PaMMA-Net predicts tokamak magnetic measurements by learning to forecast their increments rather than absolute values, using a transformer decoder and spectrogram-based data augmentation, and outperforms generic time-...
Discussion (0). Continue with ORCID to comment.