Pith. sign in

REVIEW 4 major objections 6 minor 36 references

CS-VLM: Compressed Sensing Attention for Efficient Vision-Language Representation Learning

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that attention outputs are sparse enough to be compressed by random projections and reconstructed by sparse recovery, cutting vision-language attention from quadratic to linear cost.

desk verdict The recovery guarantee doesn't survive contact with the actual computation, but the paper is a clear, motivated attempt worth engaging with. read the letter →

arxiv 2507.02957 v1 pith:7LMZBDTP submitted 2025-06-30 cs.CV

Andrew Kiruluta , Preethi Raju , Priscilla Burity This is my paper · ORCID
classification cs.CV MSC 68T0794A12
keywords compressedsensingattentionsparserecoveryvision-languagetransformerslinearRestrictedIsometryPropertycross-modalretrievalimagecaptioninglong-sequencemodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to show that attention in vision-language transformers does not need to be computed over all token pairs. Its central proposal, CSAT, compresses keys and values with random measurement matrices, computes attention in the compressed space, and reconstructs the original context vectors by sparse recovery. If this works, the quadratic $O(n^2 d)$ cost of attention drops to $O(n m d + \text{decoding})$ with $m \ll n$, which would let vision-language models process longer video and text sequences. The paper reports that this compressed model matches or slightly beats full attention and linear-attention baselines on language modeling, long-range classification, image-text retrieval, and captioning. It also claims to be the first to tie attention-output structure to sparse-signal recovery under the Restricted Isometry Property.

What carries the argument

The load-bearing object is a pair of random measurement matrices $\Phi_K, \Phi_V \in \mathbb{R}^{m \times n}$ that project the key and value matrices into $m \ll n$ dimensions before attention, together with a sparse decoder that recovers each context row from $Z_i = \Phi\Psi\alpha_i$ by $\ell^1$-minimization, approximated by ISTA or its learned unrolled form LISTA. The Restricted Isometry Property (RIP) is the condition the paper invokes to guarantee that the sparse coefficients $\alpha_i$ can be stably recovered; the complexity saving comes from computing attention scores against the $m$-dimensional compressed keys instead of all $n$ tokens.

What would settle it

Measure the decoder's reconstruction error $\|\hat{C}_i - C_i\|_2$ on tokens whose attention weights are nearly uniform over many keys; if that error rises sharply as attention entropy increases, the sparsity premise on which CSAT's guarantee rests is refuted.

Watch

Extended reading notes

Core claim

The central claim is that each attention context row $C_i$ is sparse or compressible in a dictionary $\Psi$, so that after compressing keys and values with random measurement matrices, the observed compressed row $Z_i$ is a valid compressed-sensing measurement $Z_i = \Phi\Psi\alpha_i$. The model then solves for $\alpha_i$ by $\ell^1$-minimization (basis pursuit) using ISTA or its learned unrolled form LISTA and reconstructs the context as $\hat{C}_i = \Psi\hat{\alpha}_i$. The paper asserts that this is the first formal link between attention-output structure and sparse-signal recovery under the Restricted Isometry Property, and that in practice the compressed model matches or exceeds full attention and linear-attention baselines on WikiText-103, LRA Pathfinder-X, Flickr30k retrieval, and MS-COCO captioning.

Load-bearing premise

The recovery guarantee assumes the compressed attention row is exactly a fixed random measurement matrix applied to a sparse code, but the row the network actually computes goes through a data-dependent softmax, so the fixed-measurement assumption is not established.

Editorial extensions

If this is right

  • Vision-language attention can scale linearly with sequence length without assuming the attention matrix is low-rank, because the saving comes from computing scores against $m \ll n$ compressed keys.
  • CSAT blocks can replace individual attention heads or layers in a pretrained model, giving incremental efficiency gains without retraining from scratch.
  • The sparse decoder's coefficients provide an interpretability map showing which compressed visual and textual components drive each reconstructed context vector.
  • For video and long-form inputs, where temporal and spatial redundancy is high, the efficiency gains of compressed attention should be largest.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the paper's own Section 7 concedes that dense-attention tasks may break the sparsity premise; a concrete test is whether CSAT's advantage shrinks on fine-grained video captioning or crowded-scene detection.
  • Editorial extension: the implemented measurement path includes a data-dependent softmax, so a fully rigorous version of the theory would need to fix or bound that nonlinearity; without that, the RIP guarantee is a formal analogy rather than a theorem about the trained model.
  • Editorial extension: if the sparsity premise holds, the decoder coefficients double as a per-token measure of information loss, so the measurement count $m$ could be chosen adaptively during inference, a use the paper leaves implicit.
  • Editorial extension: when the learned LISTA decoder replaces exact recovery, the theoretical guarantee degrades to the approximation quality of the unrolled network; measuring how many unrolled iterations match exact basis pursuit would quantify the trade-off.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes the Compressed Sensing Attention Transformer (CSAT), an attention mechanism that compresses the key and value matrices with random measurement matrices, computes attention scores in the compressed space, and reconstructs attention context vectors via sparse recovery (ISTA or LISTA). The authors claim a complexity reduction from O(n^2 d) to O(n m d + decoding) and assert a formal connection to compressed sensing through the Restricted Isometry Property (RIP), arguing that attention context vectors are sparse and recoverable from the compressed outputs. The experimental section reports results on WikiText-103 language modeling, LRA Pathfinder-X, and vision-language benchmarks (Flickr30k, MS-COCO), claiming competitive or slightly better performance than full attention and linear-attention baselines. The paper's novelty rests on the theoretical recovery guarantee and the modular sparse-decoder design.

Significance. If the theoretical claim were correct, the paper would introduce a principled, theoretically grounded efficient attention mechanism for vision-language models, which is a timely and valuable goal. The idea of treating attention context vectors as sparse signals and using CS-style decoding is creative, and the modular decoder (analytical vs. learned) is a practically appealing design choice. However, the central derivation is invalid: the quantity actually computed by CSAT is not a fixed linear measurement of a sparse context vector, and the main equation exhibits a dimensional mismatch. In addition, the empirical evaluation is too underspecified—no hyperparameters, no seeds, no error bars—to verify the reported results or the claimed efficiency-accuracy trade-off. The core contribution is therefore not established, and the manuscript cannot be accepted in its current form.

major comments (4)
  1. [Section 3, Eqs. (4)-(6)] The identity Z_i = ΦΨα_i does not describe the operation actually computed by the architecture. From the definitions in Section 3, Z_i = softmax(Q Φ_K^T K^T / √d_k)_i · Φ_V V, which is a nonlinear, data-dependent function of Q, K, and V. It is not a fixed linear measurement Φ_V C_i of the true context C_i = softmax(QK^T/√d_k)V, because projecting the keys before softmax changes the attention weights, and because the compressed attention row multiplies Φ_V V rather than V. Consequently, the basis-pursuit recovery guarantee with a fixed, RIP-compliant measurement matrix does not apply to CSAT as defined, and the central theoretical claim in Section 5 is unsubstantiated.
  2. [Section 3, Eq. (6)] The claimed equation Z_i = ΦΨα_i is dimensionally inconsistent. With Φ_V ∈ R^{m×n}, Ψ ∈ R^{d_k×d_k}, and α_i ∈ R^{d_k}, the product Φ_V Ψ α_i has length m, whereas Z_i is a row of the compressed attention output and therefore has length d_k. A measurement matrix for a d_k-dimensional context vector would need to act on the feature dimension (i.e., lie in R^{m×d_k}), but the paper reuses Φ_V, which acts on the token dimension. This conflates the sequence-length and feature-embedding spaces and makes the recovery problem as formulated ill-posed.
  3. [Section 3 and Eq. (9)] The claimed complexity of O(n m d + decoding) is incomplete. The decoding step is applied row-wise to Z ∈ R^{n×d_k}; for ISTA/LISTA, the per-row cost involves matrix-vector multiplications with the learned matrices S and B and depends on the number of unrolled iterations t. The paper does not specify the shapes of S and B, the value of t, or the cost per iteration, so the claimed linear scaling in n is not established. The efficiency comparison in Table 5 cannot be interpreted without a concrete complexity analysis of the decoder.
  4. [Section 4, Tables 1-4] The experimental results are reported as single point estimates with no standard deviations, number of seeds, or training details. The CSAT-specific hyperparameters m (measurement dimension), s (sparsity), and t (decoder depth) are never stated, and the vision-language experiments do not specify how the baselines were trained or fine-tuned (e.g., learning rate, batch size, number of steps, and whether the same budget was used for all methods). This makes the reported results irreproducible and the claimed efficiency-accuracy trade-off unverifiable.
minor comments (6)
  1. [Title/Abstract vs. body] The title refers to 'CS-VLM' while the architecture is called 'CSAT' throughout the text; please reconcile the nomenclature.
  2. [References] Reference [6] and reference [22] are the same paper (Papyan et al., 2020); remove the duplicate.
  3. [Section 3, Eq. (9)] The dimensions of the LISTA matrices S and B are not defined; specify their shapes in terms of m, d_k, and the decoder depth t.
  4. [Table 5] Report the hardware, batch size, and measurement methodology used for the runtime and GPU memory numbers.
  5. [Conclusion] The conclusion claims 'state-of-the-art or competitive performance,' but the results in Tables 1-4 are at best competitive with baselines; soften this claim to match the reported evidence.
  6. [Section 7] The limitations section acknowledges that the sparsity assumption may break down, but it does not address the more fundamental issue that the formal recovery guarantee is derived for a quantity that is not actually computed; please add a discussion of this gap.

Circularity Check

2 steps flagged · score 8.0 of 10

CSAT's RIP recovery guarantee is defined into the measurement equation rather than derived from the mechanism.

  1. self definitional [Section 3, Mathematical Framework (equation for Zi = ΦΨαi)]
    "Then the observed compressed output Zi can be written as: Zi = ΦΨαi, with ∥αi∥0 ≪ dk, where Φ = Φ V is reused as the measurement matrix for decoding."

    The actual computation in the same section gives Zi as the i-th row of softmax(Q Φ_K^T K^T / sqrt(dk)) · Φ_V V, a data-dependent, nonlinear function of Q, K, and V. The paper instead asserts that the observed compressed output equals a fixed linear measurement ΦΨαi of the sparse context code and then invokes standard basis-pursuit/RIP recovery for this asserted equality. The recovery guarantee is therefore an input to the framework, not a consequence of the CSAT attention mechanism. The equality also has mismatched dimensions: Zi ∈ R^{dk}, while Φ_V Ψ α_i ∈ R^m (Φ_V ∈ R^{m×n}). Hence the central theoretical claim reduces to the defining equation.

  2. self definitional [Section 5, Novelty and Contributions]
    "CSAT is the first to establish a formal connection between the structure of attention outputs and sparse signal recovery under the Restricted Isometry Property (RIP)."

    This novelty claim is load-bearing for the paper's contribution, but it is supported only by the asserted equation Zi = ΦΨαi from Section 3. Because that equation is assumed rather than derived from the actual softmax-weighted projection, the claimed 'formal connection' restates the definitional premise rather than reporting a derived or empirical discovery. The external CS theorems about RIP are real, but they are not shown to apply to the mechanism actually computed.

full rationale

The empirical benchmark comparisons (WikiText-103, LRA, Flickr30k, MS-COCO) are external and, if reproducible, would be independent evidence; they do not by themselves make the paper circular. The circularity is concentrated in the theoretical contribution: the recovery guarantee follows from an asserted equality (Zi = ΦΨαi) that is not the operation computed in Section 3, where Zi is a softmax-weighted, data-dependent projection of V through Φ_V. The paper explicitly concedes in Section 7 that sparsity of context vectors is 'the primary assumption,' which is an honest limitation but does not repair the gap: the assumption concerns sparsity, whereas the decisive step is the additional identification of the compressed output with a fixed linear measurement of the sparse code. Since the paper's claimed first formal connection to RIP is built on that identification by construction, the central derivation is circular rather than self-contained. No self-citation issues are present; all RIP/basis-pursuit references are to external, established sources.

Assumptions & free parameters 3 free parameters · 4 assumptions · 2 invented entities

The central claim rests on the assumed sparsity of attention context vectors, an unverified RIP condition on a projection that turns out to be data-dependent, the unproven equivalence between compressed-key attention and full attention, and the assertion that the computed compressed output equals a fixed measurement of a sparse vector. No public code, data, or hyperparameter settings support the benchmark numbers. The paper's Limitations section concedes several of these assumptions may fail in dense or fine-grained tasks.

free parameters (3)
  • measurement dimension m
    Number of rows in the random measurement matrices Phi_K and Phi_V; never reported in the experiments.
  • sparsity level s
    Assumed sparsity of the context coefficient vector alpha_i; no value is given and no ablation is shown.
  • decoder depth t
    Number of LISTA / ISTA iterations in the sparse decoder; listed as tunable but never specified for the reported results.
assumptions (4)
  • domain assumption Attention context vectors are exactly sparse or compressible in a fixed or learned dictionary Psi.
    Core assumption stated in Sections 1 and 3, supported only by citing neural collapse; the Limitations section concedes it may fail for dense tasks.
  • domain assumption The composed measurement Phi_V * Psi satisfies the Restricted Isometry Property with sufficient coherence guarantees.
    Standard CS condition invoked in Section 3 but never verified for the actual matrices used in the proposed architecture.
  • ad hoc to paper Attention computed against compressed keys eK approximates attention against full keys K.
    No bound or empirical study connects softmax(Q * eK^T) to softmax(Q * K^T); this is assumed for the efficiency and fidelity claim.
  • ad hoc to paper The computed compressed output Z_i equals Phi * Psi * alpha_i with a fixed measurement matrix Phi.
    By the paper's own definitions Z_i = softmax(...) * Phi_V * V; equating this to Phi_V * Psi * alpha_i requires the softmax weights to be absorbed into a fixed matrix, which is not shown.
invented entities (2)
  • Sparse dictionary Psi for attention context vectors
    purpose: Assumed basis in which each attention context vector has s-sparse coefficients, enabling compressed sensing recovery.
    No learned or fixed Psi is specified, and no evidence is provided that context vectors are sparse in any basis.
  • Random measurement matrices Phi_K and Phi_V independent evidence
    purpose: Compress keys and values before attention, reducing the attention matrix size from n x n to n x m.
    Random Gaussian / Rademacher / Hadamard matrices are standard in compressed sensing, but their specific RIP application to context vectors here is unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CS-VLM: Compressed Sensing Attention for Efficient Vision-Language Representation Learning." pith.science (2026). https://pith.science/paper/7LMZBDTP

@misc{pith2026250702957,
  author       = {Pith},
  title        = {Pith review of: CS-VLM: Compressed Sensing Attention for Efficient Vision-Language Representation Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7LMZBDTP}},
  note         = {Machine review of arXiv:2507.02957}
}
read the original abstract

Vision-Language Models (vLLMs) have emerged as powerful architectures for joint reasoning over visual and textual inputs, enabling breakthroughs in image captioning, cross modal retrieval, and multimodal dialogue. However, as these models scale to longer video sequences and richer language descriptions, the quadratic complexity of the standard attention mechanism presents a fundamental computational bottleneck. This challenge is exacerbated in vLLMs, where attention must be computed not only within modalities but also across them, leading to prohibitive memory and latency costs. In this work, we introduce the Compressed Sensing Attention Transformer (CSAT), a novel architecture that reimagines attention computation through the lens of compressed sensing. By projecting high dimensional key and value representations into a lower-dimensional subspace via random measurement matrices and reconstructing the attention outputs using sparse recovery algorithms, CSAT significantly reduces attention complexity while maintaining semantic fidelity. Applied to vLLMs, CSAT exploits the inherent compressibility of both visual and textual representations especially evident in video, where temporal redundancy is high, and in language, where cross-modal grounding is often sparse. In contrast to LLMs, which must often model entangled symbolic dependencies, vLLMs benefit from structured sparsity in alignment and scene composition, making them particularly well-suited to compressed attention. We provide a formal mathematical treatment of CSAT, demonstrate its integration into vision language pipelines, and validate its performance on standard benchmarks, highlighting its promise as a scalable, interpretable, and resource efficient solution for next generation multimodal transformers.

Figures

Figures reproduced from arXiv: 2507.02957 by the authors.

Figure 1
Figure 1. Compressed Sensing Attention Transformer (CSAT) architecture in a vision-language [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 32 canonical work pages

  1. [1]

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., and Houlsby, N. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR)

    work page 2020

  2. [2]

    Baevski, A., Zhou, Y., Mohamed, A., and Auli, M. (2020). wav2vec 2.0: A framework for self- supervised learning of speech representations. In Advances in Neural Information Processing Systems (NeurIPS), 33, 12449–12460

    work page 2020

  3. [3]

    Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., .and Hassabis, D. (2021). Highly accurate protein structure prediction with AlphaFold. Nature, 596(7873), 583–589

    work page 2021

  4. [4]

    Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The efficient transformer. In International Conference on Learning Representations (ICLR)

    work page 2020

  5. [5]

    Roy, A., Saffar, M., Vaswani, A., and Grangier, D. (2021). Efficient content-based sparse atten- tion with routing transformers. Transactions of the Association for Computational Linguistics, 9, 53–68

    work page 2021

  6. [7]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural infor- mation processing systems, 2017

    work page 2017

  7. [8]

    Compressed sensing

    David L Donoho. Compressed sensing. IEEE Transactions on Information Theory, 52(4):1289– 1306, 2006

    work page 2006

  8. [9]

    Zhang, C., Chen, X., and Lin, J. (2023). Q-CLIP: Efficient and Accurate Full-Precision Quantization for Vision-Language Models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2023

Show all 36 references
  1. [10]

    and Vinyals, O

    Jaegle, A., Borgeaud, S., Alayrac, J.-B., Doersch, C., Ionescu, C., Ding, D., . and Vinyals, O. (2021). Perceiver: General Perception with Iterative Attention. In Proceedings of the 38th International Conference on Machine Learning (ICML), pages 4651–4664. 13

    2021

  2. [11]

    J., Romberg, J

    Cand` es, E. J., Romberg, J. K., and Tao, T. (2006). Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2), 489–509

    2006

  3. [12]

    Elad, M. (2010). Sparse and Redundant Representations: From Theory to Applications in Signal and Image Processing. Springer

    2010

  4. [13]

    M., and Pauly, J

    Lustig, M., Donoho, D., Santos, J. M., and Pauly, J. M. (2007). Sparse MRI: The application of compressed sensing for rapid MR imaging. Magnetic Resonance in Medicine, 58(6), 1182–1195

    2007

  5. [14]

    F., Davenport, M

    Duarte, M. F., Davenport, M. A., Takhar, D., Laska, J. N., Sun, T., Kelly, K. F., and Bara- niuk, R. G. (2008). Single-pixel imaging via compressive sampling. IEEE Signal Processing Magazine, 25(2), 83–91

    2008

  6. [15]

    Wainwright, M. J. (2009). Sharp thresholds for high-dimensional and noisy sparsity recovery using L1-constrained quadratic programming (Lasso). IEEE Transactions on Information Theory, 55(5), 2183–2202

    2009

  7. [16]

    Sulam, J., Papyan, V., Romano, Y., and Elad, M. (2018). Multilayer convolutional sparse modeling: Pursuit and dictionary learning. IEEE Transactions on Signal Processing, 66(15), 4090–4104

    2018

  8. [17]

    Chen, Y., Liu, Y., and Gu, S. (2018). Theoretical linear convergence of unfolded ISTA and its practical weights and thresholds. Advances in Neural Information Processing Systems, 31

    2018

  9. [18]

    Wang, Z., Liu, D., Yang, J., Han, W., Huang, T., and Yu, Y. (2015). Deep networks for image super-resolution with sparse prior. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 370–378

    2015

  10. [19]

    Han, S., Pool, J., Tran, J., and Dally, W. (2015). Learning both weights and connections for efficient neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 28

    2015

  11. [20]

    Wen, W., Wu, C., Wang, Y., Chen, Y., and Li, H. (2016). Learning structured sparsity in deep neural networks. In Advances in Neural Information Processing Systems (NeurIPS), 29

    2016

  12. [21]

    Michel, P., Levy, O., and Neubig, G. (2019). Are sixteen heads really better than one? In Advances in Neural Information Processing Systems (NeurIPS), 32

    2019

  13. [22]

    Y., and Donoho, D

    Papyan, V., Han, X. Y., and Donoho, D. L. (2020). Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40), 24652–24663

    2020

  14. [23]

    Aharon, M., Elad, M., and Bruckstein, A. (2006). K-SVD: An algorithm for designing overcom- plete dictionaries for sparse representation. IEEE Transactions on Signal Processing, 54(11), 4311–4322

    2006

  15. [24]

    Calderbank, R., Jafarpour, S., and Schapire, R. (2009). Compressed learning: Universal sparse dimensionality reduction and learning in the measurement domain. Technical Report, arXiv:0901.4102

    2009 arXiv

  16. [25]

    Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information

    Emmanuel J Cand` es, Justin Romberg, and Terence Tao. Robust uncertainty principles: Exact signal reconstruction from highly incomplete frequency information. IEEE Transactions on Information Theory, 52(2):489–509, 2006. 14

    2006

  17. [26]

    S., Donoho, D

    Chen, S. S., Donoho, D. L., and Saunders, M. A. (2001). Atomic decomposition by basis pursuit. SIAM Review, 43(1), 129–159

    2001

  18. [27]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., and Sutskever, I

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., and Sutskever, I. (2021). Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML)

    2021

  19. [28]

    Li, J., Li, D., Xiong, C., and Hoi, S. C. H. (2022). BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation. In Proceedings of the 39th International Conference on Machine Learning (ICML)

    2022

  20. [29]

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hassani, A., and Zisserman, A. (2022). Flamingo: A Visual Language Model for Few-Shot Learning. arXiv preprint arXiv:2204.14198

    2022 arXiv

  21. [30]

    Chefer, H., Gur, S., and Wolf, L. (2021). Transformer Interpretability Beyond Attention Visualization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 782–791

    2021

  22. [31]

    Li, X., Yin, X., Li, C., Zhang, P., Zhang, X., Hu, X., and Wang, L. (2021). SuperVision: Su- pervised Attention for Vision-Language Tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 13658–13667

    2021

  23. [32]

    Generating long sequences with sparse transformers

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509, 2019

    1904 arXiv

  24. [33]

    Linformer: Self-attention with linear complexity

    Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768, 2020

    2006 arXiv

  25. [34]

    Rethinking attention with performers

    Krzysztof Choromanski et al. Rethinking attention with performers. In ICLR, 2021

    2021

  26. [35]

    Longformer: The long-document trans- former

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document trans- former. arXiv preprint arXiv:2004.05150, 2020

    2004 arXiv

  27. [36]

    Big bird: Transformers for longer sequences

    Manzil Zaheer et al. Big bird: Transformers for longer sequences. In NeurIPS, 2020

    2020

  28. [37]

    Learning fast approximations of sparse coding

    Karol Gregor and Yann LeCun. Learning fast approximations of sparse coding. In ICML, 2010. 15

    2010

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.