pith. sign in

arxiv: 2606.00746 · v1 · pith:Z7JFFO6Onew · submitted 2026-05-30 · 💻 cs.CV · cs.LG

Scaling Parallel Sequence Models to Foundation-Scale Vision Encoders

Yitong Jiang , Hongjun Wang , Collin McCarthy , Hanrong Ye , David Wehr , Xinhao Li , Qi Dou , Tianfan Xue
show 10 more authors
Ka Chun Cheung Simon See Wonmin Byeon Ke Chen Kai Han Jinwei Gu Hongxu Yin Pavlo Molchanov Jan Kautz Sifei Liu
This is my paper

Pith reviewed 2026-06-28 19:20 UTC · model grok-4.3

classification 💻 cs.CV cs.LG
keywords vision foundation modelsspatial propagation networkssubquadratic attention alternativescross-operator distillationCUDA kernel optimizationhigh-resolution image encodingADE20K semantic segmentation
0
0 comments X

The pith

C-GSPN scales 2D spatial propagation to foundation vision encoders, matching a ViT baseline with 15 percent fewer parameters after distillation on 600 million image-text pairs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that generalized spatial propagation networks can serve as foundation-scale vision encoders when equipped with a fused CUDA kernel, a compressed latent-space block, and a two-stage distillation process from an attention teacher. This combination preserves the 2D grid structure of images while achieving near-linear complexity, avoiding the quadratic cost of self-attention that limits resolution and pretraining scale. A reader would care because the resulting model delivers matching performance to an isomorphic vision transformer, a 2.1 percent gain on ADE20K segmentation, easier transfer to high resolutions, and a fourfold block speedup at 2K resolution with single-pass inference.

Core claim

C-GSPN shows that 2D line-scan recurrences can be made practical at foundation scale: a fast warp-specialized kernel reaches over 90 percent of peak memory bandwidth, a fused normalization block converts kernel speed into model efficiency, and cross-operator distillation on 600 million pairs transfers representational power from a full-attention teacher so that the student matches an isomorphic ViT baseline while using 15 percent fewer parameters.

What carries the argument

The C-GSPN encoder, which replaces self-attention with fused generalized spatial propagation blocks that propagate context directly on the 2D image grid through line-scan recurrences.

If this is right

  • The model improves ADE20K segmentation by 2.1 percent over the isomorphic ViT baseline.
  • High-resolution transfer requires only a fraction of the data needed for from-scratch training.
  • End-to-end block inference at 2K resolution runs four times faster with single-pass, tiling-free execution.
  • The architecture eliminates the need for positional embeddings while maintaining 2D spatial structure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same distillation recipe could be tested on other subquadratic operators to see whether 2D grid propagation generalizes beyond line scans.
  • If the kernel optimizations hold at larger batch sizes, training throughput for high-resolution vision pretraining could increase substantially.
  • The approach suggests that preserving native 2D recurrence structure may reduce the data needed for high-resolution adaptation compared with 1D token serialization methods.

Load-bearing premise

The two-stage distillation process transfers the representational power of a full attention teacher to the GSPN student at foundation scale without requiring from-scratch training or large performance loss.

What would settle it

Train an identical C-GSPN architecture from scratch on the same 600 million image-text pairs and measure whether it reaches within 1 percent of the distilled model's accuracy on downstream tasks such as ADE20K segmentation.

Figures

Figures reproduced from arXiv: 2606.00746 by Collin McCarthy, David Wehr, Hanrong Ye, Hongjun Wang, Hongxu Yin, Jan Kautz, Jinwei Gu, Ka Chun Cheung, Kai Han, Ke Chen, Pavlo Molchanov, Qi Dou, Sifei Liu, Simon See, Tianfan Xue, Wonmin Byeon, Xinhao Li, Yitong Jiang.

Figure 1
Figure 1. Figure 1: One method, C-GSPN, at two levels of efficiency. (a) System efficiency. The fast GSPN kernel turns the line scan into a single fused, warp-specialized CUDA kernel, running up to 40–52× faster than the original GSPN reference kernel across input configurations. (b) Architecture & training efficiency. Built on this fast kernel, C-GSPN’s compressed block and cross-operator distillation scale 2D spatial propag… view at source ↗
Figure 2
Figure 2. Figure 2: From the GSPN kernel to the fast GSPN kernel. (a) GSPN kernel: the original reference kernel launches a separate, lightweight kernel per image column, computing ℎ 𝑐 𝑖 = 𝜔 𝑐 𝑖 ℎ 𝑐 𝑖−1 + 𝜆 𝑐 𝑖 ⊙ 𝑥 𝑐 𝑖 and shuttling intermediate states 𝑋, 𝐺, 𝐻, 𝐵,Λ through global memory (HBM) every step, with no on-chip reuse—the bottleneck. (b) fast GSPN kernel: a single fused kernel runs the outer scan with an inner loop ov… view at source ↗
Figure 3
Figure 3. Figure 3: Step-by-step optimization of the GSPN CUDA kernel. Each bar shows the cumulative reduction in forward time (ms) from the original GSPN baseline. The final fast GSPN kernel achieves a 40.0× speedup. the shared 𝑤𝑖 play the role of an attention-style affinity matrix over positions, while the channel-specific Λ𝑗 act as value gating—so in the single-channel case the kernel is precisely an attention-like process… view at source ↗
Figure 4
Figure 4. Figure 4: C-GSPN architecture overview. (a) C-GSPN follows the ViT hierarchy of block ⊃ layer ⊃ sublayer, replacing only the attention layer. (b) The original GSPN layer operates in raw channel space and keeps the extra projections and residuals inherited from the attention template (Improvement 2’s target); C-GSPN propagates in a compressed latent space with fused row-stochastic normalization and removes the redund… view at source ↗
Figure 5
Figure 5. Figure 5: Propagation sublayer latency, original GSPN vs. C-GSPN, under increasing channels (left) and batch size (right) at 1K resolution. Original GSPN spikes as 𝐶/𝐵 grow due to GPU concurrency limits; C-GSPN’s latent-space propagation remains flat, yielding large speedups. 5.1. A More Efficient ViT Block: Latent-Space Propagation The compact-channel principle of the fast GSPN kernel (Sec. 4.2) suggests where the … view at source ↗
Figure 6
Figure 6. Figure 6: Left: Block latency vs. image resolution (𝐵=32, 𝐶=1152); the original GSPN is dominated by weight normalization and other overhead at high resolution, which C-GSPN substantially reduces. Right: Overhead reduction at 1K from removing (1) additional linear projections, (2) the inner-layer residual, and (3) channel-extension projections; cumulative speedup ≈5.5×. Executing all steps in one pass eliminates int… view at source ↗
Figure 7
Figure 7. Figure 7: Two-stage cross-operator distillation (Improvement 3). Stage 1 (sublayer-wise): each C-GSPN propagation sublayer is aligned to the teacher’s attention sublayer from the shared block input, giving a strong initialization. Stage 2 (end-to-end): the full student is distilled with two supervision taps per block—post-propagation (PP) and post-block (PB)—through lightweight feature adaptors that bridge the propa… view at source ↗
Figure 8
Figure 8. Figure 8: Runtime comparison of the original GSPN kernel and the fast GSPN kernel. Forward and backward execution times (ms) across channel counts and configurations. The fast GSPN kernel greatly improves both passes across cases. foundation vision towers: even as batch sizes reach 256 or channels reach 1024, fast GSPN sustains 2–4× speedups, e.g. a 27.4× forward and 48.6× backward speedup at 256 channels, with the … view at source ↗
Figure 9
Figure 9. Figure 9: Qualitative text-to-image results from our fast GSPN SDXL model. We enable generation up to 16K resolution on a single A100 while reducing inference time by up to 93×. 6.4. C-GSPN at Foundation Scale We distill C-GSPN from a strong attention teacher (OpenCLIP ViT-SO/14 at 378) using only 600M image–text pairs—far less than training attention models from scratch (e.g., SigLIP-v2’s 40B). Under identical data… view at source ↗
Figure 10
Figure 10. Figure 10: Latency vs. resolution. Sublayer (left) and full-block (right) latency for attention, FlashAttention, the original GSPN, and C-GSPN. Attention-based cores scale quadratically and quickly become memory- or latency-bound, while C-GSPN’s latent-space propagation stays low and resolution-stable, translating its kernel-level gains (Improvement 1) into block-level gains (Improvement 2) at 1K–2K. Sublayer latenc… view at source ↗
Figure 11
Figure 11. Figure 11: Ablations on training strategy and module structure. (a) Distillation: contrastive → +PB → +PB+PP (+adaptors); PP gives the largest gain, adaptors help, Stage-1 gives a strong start. (b) Adaptors consistently help. (c) Compression: among tested ratios, 18 gives the best observed performance–accuracy balance. (d) Hybrid: adding 3/27 attention layers improves accuracy while preserving speed. (e) Overhead tr… view at source ↗
read the original abstract

Vision foundation models are bottlenecked by the quadratic cost of self-attention, which limits usable resolution and increases the cost of large-scale pretraining. Subquadratic alternatives such as linear attention and state-space models reduce this cost, but often serialize images into 1D token streams and weaken the 2D spatial structure important for vision. Generalized Spatial Propagation Networks (GSPN) instead propagate context directly on the 2D grid through line-scan recurrences, achieving near-linear complexity without positional embeddings, but have seen little use as foundation-scale encoders. We present C-GSPN, a foundation-scale vision encoder based on 2D spatial propagation. C-GSPN makes the operator practical through three improvements: (1) a fast GSPN CUDA kernel that fuses per-step launches into a single warp-specialized implementation with shared-memory tiling, coalesced access, and a compact multi-channel propagation, reaching over 90% of peak memory bandwidth and running up to 40--52x faster than the original GSPN implementation; (2) a compressed latent-space propagation block with fused normalization, which turns kernel-level speed into block- and model-level efficiency; and (3) a two-stage cross-operator distillation recipe that trains the new architecture from an attention teacher without the cost of from-scratch foundation-scale training. Distilled with 600M image-text pairs, C-GSPN matches an isomorphic ViT baseline with 15% fewer parameters, improves ADE20K segmentation by +2.1%, transfers to high resolution with a fraction of the data needed from scratch, and delivers a 4x end-to-end block speedup at 2K with single-pass, tiling-free inference.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The paper introduces C-GSPN, a foundation-scale vision encoder based on Generalized Spatial Propagation Networks (GSPN) that propagates context on the 2D grid with near-linear complexity. It proposes three improvements—a fused CUDA kernel for the GSPN operator, a compressed latent-space propagation block with fused normalization, and a two-stage cross-operator distillation recipe from an attention teacher—trained on 600M image-text pairs. The abstract claims this yields an isomorphic ViT match with 15% fewer parameters, +2.1% ADE20K segmentation improvement, efficient high-resolution transfer, and 4x end-to-end block speedup at 2K resolution with single-pass inference.

Significance. If the distillation successfully transfers attention representations to the GSPN student at this scale without large loss, the work would offer a practical subquadratic alternative to ViT-style encoders that preserves 2D spatial structure and avoids from-scratch foundation training. The kernel and block-level optimizations are concrete engineering contributions whose correctness can be verified independently of the learning claims.

major comments (2)
  1. [Abstract] Abstract: the central performance claims (ViT matching with 15% fewer parameters, +2.1% ADE20K gain, high-res transfer) rest on the two-stage cross-operator distillation recipe, yet the manuscript supplies no definitions of the stages, loss terms, alignment objectives, or measured transfer gap on pretraining or downstream metrics. This is the load-bearing precondition for the scaling narrative.
  2. [Abstract] Abstract: the claim of successful transfer 'without the cost of from-scratch foundation-scale training' is presented without any ablation or comparison showing the performance gap between the distilled C-GSPN and a from-scratch GSPN baseline at the 600M-pair scale, leaving the efficiency of the recipe unquantified.

Simulated Author's Rebuttal

2 responses · 1 unresolved

We thank the referee for the constructive feedback on the clarity of our distillation claims and the need for supporting ablations. We address each major comment below and will revise the manuscript accordingly where feasible.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the central performance claims (ViT matching with 15% fewer parameters, +2.1% ADE20K gain, high-res transfer) rest on the two-stage cross-operator distillation recipe, yet the manuscript supplies no definitions of the stages, loss terms, alignment objectives, or measured transfer gap on pretraining or downstream metrics. This is the load-bearing precondition for the scaling narrative.

    Authors: We agree that the abstract and current manuscript text do not supply explicit definitions of the two-stage recipe, loss terms, alignment objectives, or transfer gaps. The high-level description in the abstract is insufficient as a standalone claim. In revision we will expand the methods section (new subsection 3.3) with precise definitions: Stage 1 performs supervised feature alignment via MSE on intermediate GSPN vs. attention features; Stage 2 applies end-to-end distillation using a weighted sum of KL divergence on attention maps and contrastive loss on image-text pairs. We will also report measured transfer gaps (pretraining perplexity delta and downstream metric deltas) in a new table. This directly addresses the precondition for the scaling narrative. revision: yes

  2. Referee: [Abstract] Abstract: the claim of successful transfer 'without the cost of from-scratch foundation-scale training' is presented without any ablation or comparison showing the performance gap between the distilled C-GSPN and a from-scratch GSPN baseline at the 600M-pair scale, leaving the efficiency of the recipe unquantified.

    Authors: We agree a direct from-scratch GSPN baseline at 600M pairs would better quantify the distillation efficiency. The manuscript currently provides only indirect support via smaller-scale runs and final performance matching. In revision we will add an explicit limitations paragraph stating the computational rationale and include proxy ablations at 50M-pair scale showing a 4.2% pretraining gap that narrows with distillation. Full-scale from-scratch comparison remains infeasible. revision: partial

standing simulated objections not resolved
  • Full from-scratch GSPN baseline training and direct performance-gap measurement at the 600M-pair scale (computationally prohibitive)

Circularity Check

0 steps flagged

No circularity; claims are empirical performance measurements

full rationale

The paper reports experimental outcomes from training C-GSPN via a two-stage distillation procedure on 600M image-text pairs and measuring downstream metrics against ViT baselines. No derivation chain, first-principles prediction, or fitted parameter is presented that reduces to its own inputs by construction. The CUDA kernel, compressed block, and distillation recipe are implementation and training choices whose results are validated by direct evaluation rather than by self-definition or self-citation tautology. The central scaling narrative rests on observed transfer performance, not on any equation or uniqueness theorem that collapses to prior fitted quantities.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review; no explicit free parameters, axioms, or invented entities are stated. The approach implicitly assumes standard ML training assumptions around distillation transfer and kernel correctness.

axioms (1)
  • domain assumption The described kernel, compression, and distillation changes are sufficient to make GSPN practical at foundation scale.
    The abstract presents these three improvements as the solution to prior limitations of GSPN.

pith-pipeline@v0.9.1-grok · 5903 in / 1386 out tokens · 36717 ms · 2026-06-28T19:20:11.296463+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

182 extracted references · 48 canonical work pages · 21 internal anchors

  1. [1]

    Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

    Vision mamba: Efficient visual representation learning with bidirectional state space model , author=. arXiv preprint arXiv:2401.09417 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Mamba: Linear-Time Sequence Modeling with Selective State Spaces

    Mamba: Linear-time sequence modeling with selective state spaces , author=. arXiv preprint arXiv:2312.00752 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    VMamba: Visual State Space Model

    Vmamba: Visual state space model , author=. arXiv preprint arXiv:2401.10166 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    arXiv preprint arXiv:2403.09338 , year=

    Localmamba: Visual state space model with windowed selective scan , author=. arXiv preprint arXiv:2403.09338 , year=

    work page arXiv

  5. [5]

    arXiv preprint arXiv:2403.10935 , year=

    Understanding Robustness of Visual State Space Models for Image Classification , author=. arXiv preprint arXiv:2403.10935 , year=

    work page arXiv

  6. [6]

    arXiv preprint arXiv:2309.01430 , year=

    DAT++: Spatially Dynamic Vision Transformer with Deformable Attention , author=. arXiv preprint arXiv:2309.01430 , year=

    work page arXiv

  7. [7]

    CVPR , pages=

    Cmt: Convolutional neural networks meet vision transformers , author=. CVPR , pages=

  8. [8]

    arXiv 2023 , author=

    Repvit: Revisiting mobile cnn from vit perspective. arXiv 2023 , author=. arXiv preprint arXiv:2307.09283 , year=

    work page arXiv 2023

  9. [9]

    arXiv preprint arXiv:2403.09977 , year=

    Efficientvmamba: Atrous selective scan for light weight visual mamba , author=. arXiv preprint arXiv:2403.09977 , year=

    work page arXiv

  10. [10]

    ICML , year=

    Training data-efficient image transformers & distillation through attention , author=. ICML , year=

  11. [11]

    CVPR , year=

    Imagenet: A large-scale hierarchical image database , author=. CVPR , year=

  12. [12]

    CVPR , year=

    Designing network design spaces , author=. CVPR , year=

  13. [13]

    NeurIPS , year=

    Imagenet classification with deep convolutional neural networks , author=. NeurIPS , year=

  14. [14]

    Very Deep Convolutional Networks for Large-Scale Image Recognition

    Very deep convolutional networks for large-scale image recognition , author=. arXiv preprint arXiv:1409.1556 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    CVPR , year=

    Deep Residual Learning for Image Recognition , author=. CVPR , year=

  16. [16]

    CVPR , year=

    Aggregated Residual Transformations for Deep Neural Networks , author=. CVPR , year=

  17. [17]

    CVPR , year=

    A ConvNet for the 2020s , author=. CVPR , year=

  18. [18]

    CVPR , year =

    Ding, Xiaohan and Zhang, Xiangyu and Zhou, Yizhuang and Han, Jungong and Ding, Guiguang and Sun, Jian , title =. CVPR , year =

  19. [19]

    ECCV , year=

    Microsoft COCO: Common Objects in Context , author=. ECCV , year=

  20. [20]

    Computational Visual Media , year=

    PVT v2: Improved Baselines with Pyramid Vision Transformer , author=. Computational Visual Media , year=

  21. [21]

    NeurIPS , year=

    CoAtNet: Marrying convolution and attention for all data sizes , author=. NeurIPS , year=

  22. [22]

    CVPR , year=

    MetaFormer is actually what you need for vision , author=. CVPR , year=

  23. [23]

    CVPR , year=

    Cmt: Convolutional neural networks meet vision transformers , author=. CVPR , year=

  24. [24]

    ECCV , year=

    Maxvit: Multi-axis vision transformer , author=. ECCV , year=

  25. [25]

    ICLR , year=

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale , author=. ICLR , year=

  26. [26]

    Layer Normalization

    Layer normalization , author=. arXiv preprint arXiv:1607.06450 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    ICCV , year=

    Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions , author=. ICCV , year=

  28. [28]

    CVPR , year=

    Cswin transformer: A general vision transformer backbone with cross-shaped windows , author=. CVPR , year=

  29. [29]

    NeurIPS , year=

    Attention is all you need , author=. NeurIPS , year=

  30. [30]

    arXiv preprint arXiv:2211.05778 , year=

    InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions , author=. arXiv preprint arXiv:2211.05778 , year=

    work page arXiv

  31. [31]

    arXiv preprint arXiv:2403.17695 , year=

    PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition , author=. arXiv preprint arXiv:2403.17695 , year=

    work page arXiv

  32. [32]

    NeurIPS , year=

    Focal modulation networks , author=. NeurIPS , year=

  33. [33]

    arXiv preprint arXiv:2405.07992 , year=

    MambaOut: Do We Really Need Mamba for Vision? , author=. arXiv preprint arXiv:2405.07992 , year=

    work page arXiv

  34. [34]

    arXiv preprint arXiv:2405.14174 , year=

    Multi-Scale VMamba: Hierarchy in Hierarchy Visual State Space Model , author=. arXiv preprint arXiv:2405.14174 , year=

    work page arXiv

  35. [35]

    arXiv preprint arXiv:2202.08791 , year=

    cosformer: Rethinking softmax in attention , author=. arXiv preprint arXiv:2202.08791 , year=

    work page arXiv

  36. [36]

    MMDetection: Open MMLab Detection Toolbox and Benchmark

    MMDetection: Open mmlab detection toolbox and benchmark , author=. arXiv preprint arXiv:1906.07155 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1906

  37. [37]

    MMSegmentation Contributors , howpublished =

  38. [38]

    ECCV , year=

    Deep networks with stochastic depth , author=. ECCV , year=

  39. [39]

    WACV , year=

    Efficient attention: Attention with linear complexities , author=. WACV , year=

  40. [40]

    CVPR , year=

    TransNeXt: Robust Foveal Visual Perception for Vision Transformers , author=. CVPR , year=

  41. [41]

    DINO: DETR with Improved DeNoising Anchor Boxes for End-to-End Object Detection

    Dino: Detr with improved denoising anchor boxes for end-to-end object detection , author=. arXiv preprint arXiv:2203.03605 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  42. [42]

    CVPR , year=

    Masked-attention Mask Transformer for Universal Image Segmentation , author=. CVPR , year=

  43. [43]

    Yu, Weihao and Si, Chenyang and Zhou, Pan and Luo, Mi and Zhou, Yichen and Feng, Jiashi and Yan, Shuicheng and Wang, Xinchao , journal=pami, title=

  44. [44]

    MogaNet: Multi-order Gated Aggregation Network , author=

  45. [45]

    and Feng, Jiashi and Yan, Shuicheng , title =

    Yuan, Li and Chen, Yunpeng and Wang, Tao and Yu, Weihao and Shi, Yujun and Jiang, Zi-Hang and Tay, Francis E.H. and Feng, Jiashi and Yan, Shuicheng , title =

  46. [46]

    CVPR , year=

    MViTv2: Improved multiscale vision transformers for classification and detection , author=. CVPR , year=

  47. [47]

    2024 , journal =

    Vision-LSTM: xLSTM as Generic Vision Backbone , author =. 2024 , journal =

    2024

  48. [48]

    2024 , journal =

    MambaVision: A Hybrid Mamba-Transformer Vision Backbone , author =. 2024 , journal =

    2024

  49. [49]

    2024 , journal =

    Vision-RWKV: Efficient and Scalable Visual Perception with RWKV-Like Architectures , author =. 2024 , journal =

    2024

  50. [50]

    UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning , author =

  51. [51]

    2024 , journal =

    V2M: Visual 2-Dimensional Mamba for Image Representation Learning , author =. 2024 , journal =

    2024

  52. [52]

    arXiv preprint arXiv:2402.05892 , year=

    Mamba-ND: Selective State Space Modeling for Multi-Dimensional Data , author=. arXiv preprint arXiv:2402.05892 , year=

    work page arXiv

  53. [53]

    More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity , author=

  54. [54]

    arXiv preprint arXiv:2207.05501 , year=

    Next-ViT: Next Generation Vision Transformer for Efficient Deployment in Realistic Industrial Scenarios , author=. arXiv preprint arXiv:2207.05501 , year=

    work page arXiv

  55. [55]

    Twins: Revisiting the design of spatial attention in vision transformers , author=

  56. [56]

    Swin Transformer: Hierarchical Vision Transformer using Shifted Windows , author=

  57. [57]

    Swin Transformer V2: Scaling Up Capacity and Resolution , author=

  58. [58]

    Scalable Diffusion Models with Transformers , author=

  59. [59]

    Pyramid vision transformer: A versatile backbone for dense prediction without convolutions , author=

  60. [60]

    Generating Long Sequences with Sparse Transformers

    Generating long sequences with sparse transformers , author=. arXiv preprint arXiv:1904.10509 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 1904

  61. [61]

    Lite transformer with long-short range attention , author=

  62. [62]

    Reformer: The efficient transformer , author=

  63. [63]

    Combiner: Full attention transformer with sparse computation cost , author=

  64. [64]

    Lei Zhu and Xinjiang Wang and Zhanghan Ke and Wayne Zhang and Rynson Lau , title =

  65. [65]

    Transformer-vq: Linear-time transformers via vector quantization , author=

  66. [66]

    Transformers are rnns: Fast autoregressive transformers with linear attention , author=

  67. [67]

    Rethinking attention with performers , author=

  68. [68]

    Linformer: Self-Attention with Linear Complexity

    Linformer: Self-attention with linear complexity , author=. arXiv preprint arXiv:2006.04768 , year=

    work page internal anchor Pith review Pith/arXiv arXiv 2006

  69. [69]

    Xiong, Yunyang and Zeng, Zhanpeng and Chakraborty, Rudrasis and Tan, Mingxing and Fung, Glenn and Li, Yin and Singh, Vikas , booktitle=aaai, year=. Nystr

  70. [70]

    Transformer quality in linear time , author=

  71. [71]

    Neurocomputing , year=

    Roformer: Enhanced transformer with rotary position embedding , author=. Neurocomputing , year=

  72. [72]

    arXiv preprint arXiv:2311.02077 , year=

    EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision , author=. arXiv preprint arXiv:2311.02077 , year=

    work page arXiv

  73. [73]

    Vision transformers need registers , author=

  74. [74]

    Attention is all you need , author=

  75. [75]

    Large Scale GAN Training for High Fidelity Natural Image Synthesis

    Large scale GAN training for high fidelity natural image synthesis , author=. arXiv preprint arXiv:1809.11096 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  76. [76]

    ACM SIGGRAPH 2022 conference proceedings , year=

    Stylegan-xl: Scaling stylegan to large diverse datasets , author=. ACM SIGGRAPH 2022 conference proceedings , year=

    2022

  77. [77]

    Advances in neural information processing systems , year=

    Diffusion models beat gans on image synthesis , author=. Advances in neural information processing systems , year=

  78. [78]

    Cascaded diffusion models for high fidelity image generation , author=

  79. [79]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Understanding diffusion objectives as the ELBO with simple data augmentation , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  80. [80]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=

    All are worth words: A vit backbone for diffusion models , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , year=

Showing first 80 references.