Pith. sign in

REVIEW 5 major objections 5 minor 58 references

HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HSMLA claims that applying full softmax attention only to a sparse set of task-selected spatial blocks, on top of multi-scale ReLU linear attention, recovers local detail at a fraction of the cost, delivering up to 4.2x inference-time…

desk verdict HSMLA is a sensible EfficientViT-style architecture with a clean gating idea, but the hard-threshold step that delivers the speedup is never validated and the gate definition is internally inconsistent. read the letter →

arxiv 2608.07616 v1 pith:HAA2MNCF submitted 2026-08-07 cs.CV

Dong Liu , Yanxuan Yu , Renata Borovica-Gajic , Ying Nian Wu This is my paper · ORCID
classification cs.CV
keywords visiontransformerslinearattentionsoftmaxselectivegatingdensepredictionsemanticsegmentationmedicalimageefficientinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HSMLA is designed to prove that a vision transformer can keep the global context of linear attention and regain the sharp local detail of softmax attention without paying quadratic cost everywhere. The paper's central claim is that routing full softmax computation to only a small, content-selected fraction of spatial blocks, while using ReLU-based linear attention for the rest, yields up to 4.2x inference-time speedup with equal or better accuracy on dense prediction, segmentation, super-resolution, and medical imaging. This matters because high-resolution medical and autonomous-driving inputs currently force a choice between quadratic softmax accuracy and linear-attention efficiency. The paper argues that the choice is false: the two can be combined.

What carries the argument

The load-bearing mechanism is the two-path residual kernel: a global linear-attention path with per-token output O^linear_i, and a sparse local softmax path O^local_i - O^lin-local_i that is added only where the shared block-level gate M_b exceeds threshold tau. The gate is hierarchical (head-level scores aggregated to a block-level mask), regularised by a budget term and a spatial smoothness term, and hard-thresholded at inference with about 30% of blocks selected. This yields arithmetic complexity O($Nd^{2}$ + $\alpha$ N $w^{2}$ d), with the softmax window w << $\sqrt$(N), so global context stays linear in token count and softmax sharpening is confined to a constant-size neighbourhood.

What would settle it

Take the trained checkpoint and validation images, compute the soft-gated output of Eq. (20) and the hard-threshold output of Eq. (21), and compare accuracy: if the hard-threshold accuracy drops more than a small margin, the training-inference gap claim fails. Alternatively, inspect whether organ boundaries and tumor margins in BTCV and CAMELYON16 fall inside the selected blocks at the reported threshold; if they are systematically excluded, the gate has not learned what the argument requires.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that local softmax attention can be treated as a sparse residual correction on top of a global linear attention backbone, rather than as an alternative that must be applied to every token. HSMLA computes the linear attention output for all tokens using multi-scale depthwise-convolved ReLU features, then learns a block-level gate mask M_b from task supervision that marks roughly 30% of spatial tiles as needing refinement. At inference, only those tiles run a local w x w softmax, and the output is the linear path plus the difference between local softmax and locally-restricted linear attention. The paper reports that this preserves or improves accuracy while cutting latency to 19.5 ms on Cityscapes (81.8% mIoU, 3.7x over SegFormer-B2), 87.3% Dice with 3.2x speedup on CT organ segmentation, and 94.2% AUC with 4.1x speedup on pathology whole-slide images.

Load-bearing premise

The load-bearing assumption is that a learned per-block score reliably identifies the small set of image regions that need softmax attention, and that replacing the soft gate used during training with a fixed cutoff at inference does not cost meaningful accuracy.

Editorial extensions

If this is right

  • High-resolution dense prediction can run 3 to 4 times faster than dense softmax transformers at the same input size without sacrificing accuracy.
  • Medical imaging tasks that depend on sharp boundaries, such as organ segmentation, tumor detection, and nodule detection, gain the most because softmax is routed to exactly those regions.
  • The wall-clock speedup transfers across tasks: segmentation, classification, super-resolution, and pathology whole-slide imaging all show speedups between 2.3x and 4.2x.
  • Models can run on memory-constrained hardware: surgical instrument segmentation on a Jetson-class GPU drops memory from 3.7 GB to 2.1 GB while keeping real-time frame rates.
  • On the Cityscapes Pareto frontier, HSMLA-B2 simultaneously improves accuracy and reduces latency relative to the multi-scale linear attention baseline it extends.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mechanism generalizes as the paper suggests, the same block-level gate could select volumetric or spatiotemporal tiles in 3D and video transformers, likely preserving the speedup for CT volumes and video streams.
  • The learned gate mask is itself a side product: an interpretable map of where the model believes local sharpening is needed, which could be compared against radiologist annotations or used as a free saliency signal.
  • A natural testable extension is adaptive sparsity: instead of a fixed threshold, let the gating module predict per-image budgets so that complex images spend more softmax compute than simple ones, tightening the accuracy-latency trade-off further.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes HSMLA, a transformer attention module that combines ReLU-based linear attention for global context with a learned block-level gate that routes roughly 30% of spatial blocks to a local softmax refinement path, together with multi-scale depthwise convolutions and a sandwich FFN-DWConv block. The manuscript reports accuracy and latency results on semantic segmentation, classification, super-resolution, and medical imaging benchmarks, claiming up to 4.2x inference-time speedup with equal or better accuracy than dense softmax baselines. The central claim is that selective softmax refinement via a task-learned gate preserves accuracy while the linear backbone provides efficiency.

Significance. If the results hold, HSMLA would be a practically useful contribution to efficient high-resolution dense prediction, particularly for medical imaging where boundary sharpness matters. The two-path residual formulation in Eqs. (18)-(21) is conceptually clean, and the complexity accounting O(Nd^2) + O(alpha N w^2 d) in Eq. (29) is standard and internally consistent. The component ablation in Table 2 provides some support for the role of multi-scale linear attention and sparse softmax gating. However, several load-bearing points are not yet established: the training-to-inference binarization gap of the gate is unmeasured, the gate definition is inconsistent across equations, the ablation latency numbers are counterintuitive, and the central latency claims are not reproducible without code or kernel releases.

major comments (5)
  1. [§3.3, Eqs. (20)-(21)] The hard-threshold binarization of the learned gate is not validated. Training uses the soft gate M_b in Eq. (20), while inference replaces it with 1[M_b > tau] at tau=0.15 in Eq. (21). The text asserts that the soft residual 'converges smoothly' to hard routing, but no experiment reports the soft-gate evaluation accuracy, the hard-gate accuracy on the same checkpoint, or the gate precision/recall against organ boundaries or tumor masks. Since the paper's accuracy-speedup tradeoff depends on binarization preserving the soft-gated training behavior, this is a load-bearing omission. The authors should report: (1) inference with the soft gate kept as a multiplier, (2) hard-thresholded accuracy on the same checkpoint, and (3) a sweep of tau showing how alpha and accuracy vary.
  2. [§3.2, Eqs. (11)-(12), Algorithm 1] The gating mechanism is defined inconsistently. Eq. (11) computes head-level scores from the multi-scale per-head features [Qhat_b; Khat_b; Vhat_b], but Eq. (12) asserts the shared block gate equals sigma(AvgPool(Conv3x3(X_b))), and Algorithm 1 (line 7) uses only X_b. No derivation shows how the head-level aggregation in Eq. (11) reduces to the Conv3x3(X_b) form. If the gate truly sees only the input X_b, then claims that gating is guided by multi-scale attention features, and that it localizes organ boundaries and tumor regions, are not supported by the architecture as written. The authors need to clarify which quantity is used and ensure Eq. (11), Eq. (12), and Algorithm 1 are mutually consistent.
  3. [Table 2, §4.2] The component ablation contains a physically counterintuitive result: adding SSG to 'MSLA + sandwich' reduces latency from 27.8 ms to 19.5 ms, even though SSG adds a gating computation and a local softmax pass over roughly 30% of blocks. Under the same backbone dimensions, adding computation should increase latency, not decrease it. This suggests the rows differ in more than the ablated component (e.g., channel widths, number of layers, or input resolution). The authors should provide a controlled comparison where only the SSG component is toggled, with identical architecture settings, and report the latency of the gating and refinement kernels separately.
  4. [§4.1, §3.4] The inference-speedup claims cannot be verified without code or kernel implementation. Section 3.4 and Algorithm 1 describe a custom tile-compaction pipeline, prefix-sum based dense lists, and Tensor-Core local softmax kernels, but none of this is released. The latency numbers in Tables 1, 3, 4, and 5 are therefore unreproducible. The authors should release code and kernels, or at minimum provide detailed profiler measurements (per-stage latency, memory traffic, occupancy) for both HSMLA and each baseline under identical inference settings, including exact TensorRT/PyTorch versions, precision, and batch size.
  5. [Tables 1, 2, 5; Figure 3] Several accuracy and latency claims lack uncertainty estimates. Table 1 reports mIoU differences as small as +0.3 to +1.5 points over baselines without error bars or multiple seeds, and Table 5 reports PSNR differences of 0.01-0.14 dB without variance. The Pareto-frontier claim in Figure 3 rests on these fine-grained differences. The authors should report standard deviations over at least three runs for non-medical tables as well, or temper the Pareto claim to the level of precision the data support.
minor comments (5)
  1. [§4.2, Eq. (14)] The text after Table 2 says the mask selects roughly alpha=0.3 of tokens, but Eq. (14) defines alpha as the fraction of blocks, not tokens; please use consistent terminology.
  2. [§3.2, Eq. (12)] The notation 'AvgPool' and 'Conv3x3' is under-specified: please state the pooling kernel/stride, the convolution channel dimensions, and how the output is reshaped to one scalar per block.
  3. [§3.3, Eq. (17)] The definition of O^{lin-local}_i uses phi(Qhat_i), but Eq. (16) uses raw Q_i; please clarify whether the local linear path operates on transformed or raw queries and keys.
  4. [§4.4, Table 4] The memory measurements are reported as single values with uncertainties, but the text does not describe how memory is measured (peak CUDA memory? PyTorch cache?); please add a sentence on the measurement methodology.
  5. [Figure 2] The color legend (red softmax, yellow gated, blue linear) appears only in the text; please add it to the figure caption for self-containedness.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported accuracy and speedup come from external benchmark measurements, not from a self-referential derivation.

full rationale

The paper's central claims are empirical: accuracy figures (mIoU, Dice, AUC, PSNR) and latency speedups are measured against external baselines (SegFormer, Swin, EfficientViT, Restormer), not derived from the gate equations. The gating mask M_b is a learned function of input features (Eq. 12) trained with a task loss plus budget and smoothness regularizers (Eq. 23), and the inference rule (Eq. 21) is presented as the hard-threshold version of the same residual decomposition used in training (Eq. 20); this is a consistency statement about train/inference behavior, not an identity that forces the reported accuracy. The authors' self-citations (Refs. 23-29, 51) appear only in the related-work discussion of caching and efficient biomedical designs and are not load-bearing for HSMLA's design or results. The closest concern is that the train-to-inference binarization gap is asserted rather than experimentally measured, but that is an empirical validity risk, not a logical circularity. No fitted parameter is renamed as a prediction, and no equation reduces to its own output.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central method rests on standard linear-attention algebra plus a learned gate whose hard/soft transfer is assumed. No new physical entities are introduced. The main free parameters are the threshold and regularizer weights that set the speed-accuracy operating point.

free parameters (4)
  • Gating threshold tau = 0.15 (alpha about 0.3)
    Hand-chosen in Sections 3.2 and 4.2 to select roughly 30 percent of blocks for softmax at inference; it directly sets the speed-accuracy operating point.
  • Gate regularizer weights lambda1 and lambda2 = 0.01 and 0.005
    Constants in Eq. (23) that balance the softmax budget and spatial smoothness terms; ablated but not tuned per task.
  • Target softmax budget rho = not reported
    A learnable scalar in Eq. (23) intended to stabilize the expected softmax fraction, but its final value is not reported in the paper.
  • Local window size w and block size B = w in {7,11}; B not specified
    Architecture hyperparameters that determine the local softmax neighborhood and gating tile size; they appear in the complexity term O(alpha N w^2 d).
assumptions (4)
  • standard math ReLU-based linear attention can be computed associatively so the global sums Z and D are reused for all queries.
    Invoked in Section 3.1, Eqs. (8) through (10), following Katharopoulos et al.
  • domain assumption Linear attention produces flatter attention distributions and therefore weakens local context modeling.
    Stated in the Introduction and Section 2.2 with citations [36, 48]; this motivates the entire softmax refinement path.
  • ad hoc to paper A gate mask learned from task supervision marks the spatial blocks where softmax correction is beneficial, and hard thresholding at inference preserves the soft-gated training behavior.
    Central to Eqs. (12), (20), and (21). The paper asserts the training-inference gap narrows but does not prove or analyze it.
  • domain assumption Latency comparisons across baselines are fair because all models run on the same GPU with batch size 1 and similar settings.
    Section 4.1 states this, but no code or measurement scripts are provided to verify that the custom kernels did not receive favorable treatment.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers." pith.science (2026). https://pith.science/paper/HAA2MNCF

@misc{pith2026260807616,
  author       = {Pith},
  title        = {Pith review of: HSMLA: Hierarchical Softmax Multi-scale Linear Attention for Efficient Vision Transformers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HAA2MNCF}},
  note         = {Machine review of arXiv:2608.07616}
}
abstract

Vision transformers face significant computational overheads in high-resolution dense prediction due to the quadratic complexity of self-attention. Linear attention offers efficiency but sacrifices local context modeling. We propose \textbf{HSMLA (Hierarchical Softmax Multi-scale Linear Attention)}, which combines ReLU-based linear attention for global context, selective softmax refinement for critical local features, and multi-scale token representations via depthwise convolutions. HSMLA achieves superior accuracy-efficiency trade-offs: up to $4.2\times$ inference-time speedup across dense prediction tasks, $87.3%$ Dice with $3.2\times$ speedup on CT organ segmentation, and $94.2%$ AUC with $4.1\times$ speedup on pathology WSI.

Figures

Figures reproduced from arXiv: 2608.07616 by the authors.

Figure 1
Figure 1. HSMLA architecture: (a) attention pattern comparison; (b) block design. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Attention pattern comparison across 7 methods and 6 ROI positions. HSMLA (bot [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Accuracy and runtime comparison of HSMLA against baseline models. HSMLA [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study and detailed efficiency analysis of HSMLA components on [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

58 extracted references · 36 canonical work pages

  1. [1]

    2017 robotic instrument segmentation challenge.arXiv preprint arXiv:1902.06426, 2019

    Max Allan, Alex Shvets, Thomas Kurmann, Zichen Zhang, Rahul Duggal, Yun-Hsuan Su, Nicola Rieke, Iro Laina, Nikhil Kalavakonda, Sebastian Bodenstedt, et al. 2017 robotic instrument segmentation challenge.arXiv preprint arXiv:1902.06426, 2019

    arXiv 2017

  2. [2]

    Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.JAMA, 318(22):2199–2210, 2017

    Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes van Diest, Bram van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen A W M van der Laak, and the CAMELYON16 Consortium. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer.JAMA, 318(22):2199–2210, 2017

    work page 2017

  3. [3]

    Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer.arXiv preprint arXiv:2004.05150, 2020

    arXiv 2004

  4. [4]

    Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction

    Han Cai, Junyan Li, Muyan Hu, Chuang Gan, and Song Han. Efficientvit: Lightweight multi-scale attention for high-resolution dense prediction. InProceedings of the IEEE/CVF international conference on computer vision, pages 17302–17313, 2023

    2023

  5. [5]

    Swin-UNet: Unet-like pure transformer for medical image segmenta- tion

    Hu Cao, Yueyue Wang, Joy Chen, Dongsheng Jiang, Xiaopeng Zhang, Qi Tian, and Manning Wang. Swin-UNet: Unet-like pure transformer for medical image segmenta- tion. InEuropean Conference on Computer Vision Workshops, pages 205–218, 2022

    work page 2022

  6. [6]

    TransUNet: Transformers make strong encoders for medical image segmentation.arXiv preprint arXiv:2102.04306, 2021

    Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. TransUNet: Transformers make strong encoders for medical image segmentation.arXiv preprint arXiv:2102.04306, 2021

    arXiv 2021

  7. [7]

    Re- thinking atrous convolution for semantic image segmentation, 2017

    Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Re- thinking atrous convolution for semantic image segmentation, 2017. URLhttps: //arxiv.org/abs/1706.05587

    arXiv 2017

  8. [8]

    Xuanyao Chen, Zhijian Wang, Haotian Zhou, and Song Han. Sparsevit: Revisiting activation sparsity for efficient high-resolution vision transformer.Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2061–2070, 2023

    work page 2023

Show all 58 references
  1. [9]

    Recursive generalization transformer for image super-resolution.arXiv preprint arXiv:2303.06373, 2024

    Zheng Chen, Yulun Zhang, Jinjin Gu, Yongbing Zhang, Linghe Kong, and Xiaokang Yuan. Recursive generalization transformer for image super-resolution.arXiv preprint arXiv:2303.06373, 2024

    2024 arXiv

  2. [10]

    Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers.arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  3. [11]

    Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Georgiana-Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Q

    Krzysztof M. Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Georgiana-Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Q. Davis, Afroz Mohi- uddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, and Adrian Weller. Rethinking attention with performers. InIntern...

    2021

  4. [12]

    The cityscapes dataset for semantic urban scene understanding

    Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. InProceedings of the IEEE conference on computer vision and pattern recog...

    2016

  5. [13]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009

    2009

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Syl- vain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXi...

    2010 arXiv

  7. [15]

    Segnext: Rethinking convolutional attention design for semantic segmen- tation.Advances in neural information processing systems, 35:1140–1156, 2022

    Meng-Hao Guo, Cheng-Ze Lu, Qibin Hou, Zhengning Liu, Ming-Ming Cheng, and Shi-Min Hu. Segnext: Rethinking convolutional attention design for semantic segmen- tation.Advances in neural information processing systems, 35:1140–1156, 2022

    2022

  8. [16]

    Re- conFormer: Accelerated MRI reconstruction using recurrent transformer.IEEE Trans

    Pengfei Guo, Yiqun Mei, Jinyuan Zhou, Shanshan Jiang, and Vishal M Patel. Re- conFormer: Accelerated MRI reconstruction using recurrent transformer.IEEE Trans. Med. Imaging, 43(1):582–593, 2024

    2024

  9. [17]

    Fastervit: Fast vision transformers with hierarchical attention.arXiv preprint arXiv:2306.06189, 2023

    Ali Hatamizadeh, Greg Heinrich, Hongxu Yin, Andrew Tao, Jose M Alvarez, Jan Kautz, and Pavlo Molchanov. Fastervit: Fast vision transformers with hierarchical attention.arXiv preprint arXiv:2306.06189, 2023

    2023 arXiv

  10. [18]

    Trans- formers are rnns: Fast autoregressive transformers with linear attention

    Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Trans- formers are rnns: Fast autoregressive transformers with linear attention. InInterna- tional conference on machine learning, pages 5156–5165. PMLR, 2020

    2020

  11. [19]

    Reformer: The efficient trans- former.arXiv preprint arXiv:2001.04451, 2020

    Nikita Kitaev, Łukasz Kaiser, and Anselm Levskaya. Reformer: The efficient trans- former.arXiv preprint arXiv:2001.04451, 2020

    2001 arXiv

  12. [20]

    Langerak, and Arno Klein

    Bennett Landman, Zhoubing Xu, Juan Eugenio Iglesias, Martin Styner, Thomas R. Langerak, and Arno Klein. 2015 MICCAI multi-atlas labeling beyond the cranial vault – workshop and challenge. Synapse dataset and workshop materials, 2015

    2015

  13. [21]

    Efficientformer: Vision transformers at mobilenet speed

    Yanyu Li, Geng Yuan, Yang Wen, Eric Hu, Georgios Evangelidis, Sergey Tulyakov, Yanzhi Wang, and Jian Ren. Efficientformer: Vision transformers at mobilenet speed. Advances in Neural Information Processing Systems, 35:12934–12949, 2022

    2022

  14. [22]

    Not all patches are what you need: Expediting vision transformers via token reorgani- zations

    Youwei Liang, Chongjian Ge, Zhan Tong, Yibing Song, Jue Wang, and Pengtao Xie. Not all patches are what you need: Expediting vision transformers via token reorgani- zations. InInternational Conference on Learning Representations, 2022

    2022

  15. [23]

    Tinyserve: Query-aware cache selection for efficient llm serving

    Dong Liu and Yanxuan Yu. Tinyserve: Query-aware cache selection for efficient llm serving. InProceedings of the 33rd ACM International Conference on Multimedia, pages 12529–12537, 2025

    2025

  16. [24]

    Pikv: Kv cache manage- ment system for mixture of experts.arXiv preprint arXiv:2508.06526, 2025

    Dong Liu, Yanxuan Yu, Ben Lengerich, and Ying Nian Wu. Pikv: Kv cache manage- ment system for mixture of experts.arXiv preprint arXiv:2508.06526, 2025

    2025 arXiv

  17. [25]

    Fast- cache: Fast caching for diffusion transformer through learnable linear approximation

    Dong Liu, Yanxuan Yu, Jiayi Zhang, Yifan Li, Ben Lengerich, and Ying Nian Wu. Fast- cache: Fast caching for diffusion transformer through learnable linear approximation. arXiv preprint arXiv:2505.20353, 2025. LIU ET AL.: HSMLA17

    2025

  18. [26]

    To keep or not to keep: Learning KV cache retention in disaggregated LLM serving systems

    Dong Liu, Yanxuan Yu, Eric Jiang, Shu Wang, and Ying Nian Wu. To keep or not to keep: Learning KV cache retention in disaggregated LLM serving systems. InPro- ceedings of the 19th ACM International Systems and Storage Conference, 2026

    2026

  19. [27]

    Adacorrection: Adap- tive offset cache correction for accurate diffusion transformers.arXiv preprint arXiv:2602.13357, 2026

    Dong Liu, Yanxuan Yu, Ben Lengerich, and Ying Nian Wu. Adacorrection: Adap- tive offset cache correction for accurate diffusion transformers.arXiv preprint arXiv:2602.13357, 2026

    2026 arXiv

  20. [28]

    Mka: Memory-keyed attention for efficient long-context reasoning

    Dong Liu, Yanxuan Yu, Ben Lengerich, and Ying Nian Wu. Mka: Memory-keyed attention for efficient long-context reasoning. InProceedings of the 23rd ACM Inter- national Conference on Computing Frontiers, pages 280–289, 2026

    2026

  21. [29]

    Accelerating frequency domain diffusion models with error-feedback event-driven caching.arXiv preprint arXiv:2604.22901, 2026

    Dong Liu, Yanxuan Yu, and Ying Nian Wu. Accelerating frequency domain diffusion models with error-feedback event-driven caching.arXiv preprint arXiv:2604.22901, 2026

    2026 arXiv

  22. [30]

    Vmamba: Visual state space model.arXiv preprint arXiv:2401.10166, 2024

    Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qix- iang Ye, and Yunfan Liu. Vmamba: Visual state space model.arXiv preprint arXiv:2401.10166, 2024

    2024 arXiv

  23. [31]

    Swin transformer: Hierarchical vision transformer using shifted win- dows

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted win- dows. InProceedings of the IEEE/CVF international conference on computer vision, pages 10012–10022, 2021

    2021

  24. [32]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11976–11986, 2022

    2022

  25. [33]

    MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer

    Sachin Mehta and Mohammad Rastegari. MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer. InInternational Conference on Learning Representations (ICLR), 2022. URLhttps://openreview.net/forum?id= qUcX0Zn5ROG

    2022

  26. [34]

    Adavit: Adaptive vision transformers for efficient image recogni- tion

    Lingchen Meng, Hengduo Li, Bor-Chun Chen, Shiyi Lan, Zuxuan Wu, Yu-Gang Jiang, and Ser-Nam Lim. Adavit: Adaptive vision transformers for efficient image recogni- tion. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12309–12318, 2022

    2022

  27. [35]

    Online normalizer calculation for softmax

    Maxim Milakov and Natalia Gimelshein. Online normalizer calculation for softmax. arXiv preprint arXiv:1805.02867, 2018

    2018 arXiv

  28. [36]

    Random feature attention.arXiv preprint arXiv:2103.02143, 2021

    Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah A Smith, and Ling- peng Kong. Random feature attention.arXiv preprint arXiv:2103.02143, 2021

    2021 arXiv

  29. [37]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. InAd- vances in neural information processing systems, volume 34, pages 13937–13949, 2021. 18LIU ET AL.: HSMLA

    2021

  30. [38]

    Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas de Bel, Moira S N Berens, Cas van den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al. Validation, comparison, and combination of algorithms for auto- matic detection of pulmonary...

    2017

  31. [39]

    Efficient attention: Attention with linear complexities

    Zhuoran Shen, Mingyuan Zhang, Haiyu Zhao, Shuai Yi, and Hongsheng Li. Efficient attention: Attention with linear complexities. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3531–3539, 2021. URLhttps://openaccess.thecvf.com/con...

    2021

  32. [40]

    Ntire 2017 challenge on single image super-resolution: Methods and results

    Radu Timofte, Eirikur Agustsson, Luc Van Gool, Ming-Hsuan Yang, and Lei Zhang. Ntire 2017 challenge on single image super-resolution: Methods and results. InPro- ceedings of the IEEE conference on computer vision and pattern recognition work- shops, pages 114–125, 2017

    2017

  33. [41]

    Training data-efficient image transformers & distillation through attention

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablay- rolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. InInternational conference on machine learning, pages 10347– 10357. PMLR, 2021

    2021

  34. [42]

    Med- ical transformer: Gated axial-attention for medical image segmentation

    Jeya Maria Jose Valanarasu, Pooja Oza, Ilker Hacihaliloglu, and Vishal M Patel. Med- ical transformer: Gated axial-attention for medical image segmentation. InMICCAI, pages 36–46, 2021

    2021

  35. [43]

    HAT: Hardware-aware transformers for efficient natural language process- ing

    Hanrui Wang, Zhanghao Wu, Zhijian Liu, Han Cai, Ligeng Zhu, Chuang Gan, and Song Han. HAT: Hardware-aware transformers for efficient natural language process- ing. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7675–7688, Onlin...

    2020 doi

  36. [44]

    SpAtten: Efficient sparse attention ar- chitecture with cascade token and head pruning

    Hanrui Wang, Zhekai Zhang, and Song Han. SpAtten: Efficient sparse attention ar- chitecture with cascade token and head pruning. In2021 IEEE International Sym- posium on High-Performance Computer Architecture (HPCA), pages 97–110. IEEE,

  37. [45]

    Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity.arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  38. [46]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions

    Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. InProceedings of the IEEE/CVF international conference on computer vision, pag...

    2021

  39. [47]

    Segformer: Simple and efficient design for semantic segmentation with trans- formers.Advances in neural information processing systems, 34:12077–12090, 2021

    Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with trans- formers.Advances in neural information processing systems, 34:12077–12090, 2021. LIU ET AL.: HSMLA19

    2021

  40. [48]

    Nyströmformer: A nyström-based algorithm for approximat- ing self-attention.Proceedings of the AAAI conference on artificial intelligence, 35 (16):14138–14148, 2021

    Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer: A nyström-based algorithm for approximat- ing self-attention.Proceedings of the AAAI conference on artificial intelligence, 35 (16):14138–14148, 2021

    2021

  41. [49]

    Evo-vit: Slow-fast token evolution for dynamic vision transformer

    Yifan Xu, Zhijie Zhang, Mengdan Zhang, Kekai Sheng, Ke Li, Weiming Dong, Liqing Zhang, Changsheng Xu, and Xing Sun. Evo-vit: Slow-fast token evolution for dynamic vision transformer. InProceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2964–2972, 2022

    2022

  42. [50]

    A-vit: Adaptive tokens for efficient vision transformer

    Hongxu Yin, Arash Vahdat, Jose M Alvarez, Arun Mallya, Jan Kautz, and Pavlo Molchanov. A-vit: Adaptive tokens for efficient vision transformer. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10809– 10818, 2022

    2022

  43. [51]

    Coprimeeeg: Crt-guided dual-branch reconstruction from co-prime sub-nyquist eeg

    Yanxuan Yu, Dong Liu, and Ying Nian Wu. Coprimeeeg: Crt-guided dual-branch reconstruction from co-prime sub-nyquist eeg. In2026 IEEE 23rd International Sym- posium on Biomedical Imaging (ISBI), pages 1–5. IEEE, 2026

    2026

  44. [52]

    Big bird: Transformers for longer sequences.Advances in neural information process- ing systems, 33:17283–17297, 2020

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Al- berti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences.Advances in neural information process- ing systems, 33:17283–17297, 2020

    2020

  45. [53]

    Restormer: Efficient transformer for high-resolution image restoration

    Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang. Restormer: Efficient transformer for high-resolution image restoration. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5728–5739, 2022

    2022

  46. [54]

    Pyramid scene parsing network

    Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 2881–2890, 2017

    2017

  47. [55]

    PSANet: Point-wise spatial attention network for scene parsing

    Hengshuang Zhao, Yi Zhang, Shu Liu, Jianping Shi, Chen Change Loy, Dahua Lin, and Jiaya Jia. PSANet: Point-wise spatial attention network for scene parsing. In Proceedings of the European Conference on Computer Vision (ECCV), pages 267–283, 2018

    2018

  48. [56]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Tor- ralba. Scene parsing through ade20k dataset. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017

    2017

  49. [57]

    Biformer: Vision transformer with bi-level routing attention.Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 10323–10333, 2023

    Lei Zhu, Xinjiang Wang, Zhanghan Ke, Wayne Zhang, and Rynson Lau. Biformer: Vision transformer with bi-level routing attention.Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 10323–10333, 2023

    2023

  50. [2021]

    URLhttps://doi.org/10

    doi: 10.1109/HPCA51647.2021.00018. URLhttps://doi.org/10. 1109/HPCA51647.2021.00018

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.