Pith. sign in

REVIEW 5 major objections 5 minor 54 references

MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Cluster-masked and multi-task pretraining raises xLSTM vision to 83.4% top-1 on ImageNet-1K.

desk verdict First xLSTM autoregressive vision pretraining with modest gains, but the headline result appears to come from an inpainting variant not the claimed cluster-AR objective. read the letter →

arxiv 2412.10730 v1 pith:JBQCOQ2B submitted 2024-12-14 cs.CV

Wenjun Huang , Jianguo Hu This is my paper · ORCID
classification cs.CV
keywords xLSTMautoregressivepretrainingcluster-maskedmaskingmulti-tasklearningvisualrepresentationImageNet-1Ksemanticsegmentationmaskedmodeling
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

xLSTM is an extended LSTM architecture with exponential gating and parallel matrix memory that can be turned into a vision backbone, but the paper argues its visual potential has not been realized because it has mostly been trained with standard supervised recipes. MAL is a two-stage pretraining framework designed to fix this: first, an autoregressive stage that predicts clusters of patches rather than individual pixels or patches, using a causal content mask so each cluster attends only to earlier clusters; second, a multi-task stage that adds depth estimation and semantic segmentation to the same encoder-decoder setup. The paper reports that MAL-B reaches 83.4% top-1 on ImageNet-1K, compared with 82.4% for the supervised ViL-B baseline and 81.9% for Vim-B, and that MAL-S improves ADE20K segmentation mIoU by 1.2 over ViL-S. If these results hold, the paper has found a self-supervised pretraining pathway that lets xLSTM backbones scale and compete with attention- and Mamba-based vision models rather than lagging behind them.

What carries the argument

The load-bearing mechanism is the cluster-masked autoregressive pretraining scheme, driven by a causal content mask over the serialized image: position $i$ is allowed to attend only to itself and earlier positions, with forbidden attention set to $-\infty$ in the attention logits. The prediction unit is a spatial cluster $c_i\in\mathbb{R}^{H_c\times W_c}$ formed by grouping adjacent $16\times16$ patches (default $4\times4$ clusters), which shortens the sequence and supplies a stronger local target than pixels or single patches. Around this, MAL alternates mLSTM (matrix-memory LSTM) blocks that scan the sequence forward from the top left and backward from the bottom right in the xLSTM encoder, and a separate transformer decoder with its own parameters reconstructs the input. The second stage attaches task-specific linear heads for depth and segmentation while keeping the same encoder-decoder, then discards the decoder at fine-tuning.

What would settle it

Run the MAL-B pretraining pipeline with the cluster size and masking ratio selected on a separate validation split (or a fixed rule chosen before seeing ImageNet-1K), then compare with ViL-B under identical fine-tuning; if the resulting top-1 is no better than ViL-B's 82.4%, the cluster-masked and multi-task pretraining claim would not survive an unbiased estimate. Alternatively, keep the default 9-unit/20% configuration but replace cluster targets with 1x1 patch targets and identical compute; if the gap between 83.4% and the patch-unit result disappears, the cluster mechanism, not the two-stage multitask schedule, is the source of the gain.

Watch

Extended reading notes

Core claim

The central claim is that the quality of autoregressive visual pretraining for xLSTM is governed less by the backbone and more by what the model is asked to predict and by auxiliary tasks. Grouping spatially adjacent patches into clusters as prediction units, with a cluster size of 4x4 patches (9 prediction units per 144-token sequence), outperforms both pixel- and patch-level units; the paper's ablations show 83.4% top-1 with 9 units versus 81.9% with 144 patch units and a 81.5% supervised baseline. A second pretraining stage that jointly performs autoregression, depth estimation, and segmentation raises accuracy further, from 82.5% to 83.4% for MAL-B, even though only the encoder is kept during fine-tuning and task heads are replaced by a linear classifier. The conclusion the paper draws is that cluster-masked and multi-task pretraining fully leverages the scaling potential of xLSTM and surpasses traditional supervised models.

Load-bearing premise

The reported headline accuracy (83.4% for the base-size MAL model) is the result of choosing the cluster size and masking ratio by looking at ImageNet-1K validation accuracy, then reporting that chosen configuration against fixed baselines; if that selection is not accounted for, the claimed gain over ViL-B may be smaller than it appears.

Editorial extensions

If this is right

  • MAL-B's 83.4% top-1 on ImageNet-1K would be the strongest reported result among xLSTM, Mamba, and attention-based vision backbones at base scale in the paper's comparison, beating ViL-B by 1.0 point and Vim-B by 1.5 points.
  • Replacing pixel- and patch-level prediction units with 4x4 clusters improves the same encoder's top-1 accuracy from 81.9% to 83.4%, making cluster-masked autoregression the recommended pretraining target for vision xLSTM.
  • A second pretraining stage with depth and segmentation tasks raises MAL-B from 82.5% to 83.4%, even though only the encoder and a linear classification head are kept for fine-tuning.
  • The same pretrained encoder transfers to ADE20K semantic segmentation, with MAL-S improving mIoU from 46.3 to 47.5 over ViL-S under the UperNet framework.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would expect cluster-masked targets to transfer to other causal vision sequence models, since the mechanism is defined at the serialization level rather than inside xLSTM; swapping xLSTM for Mamba or linear attention would isolate where the gain comes from.
  • Because the cluster size and masking ratio were selected by maximizing ImageNet-1K validation accuracy, I would treat the reported 83.4% as an optimistic estimate until the configuration is fixed on a separate split or chosen as a default before seeing the test set.
  • The +0.9-point multi-task gain comes from only two auxiliary datasets (NYU Depth v2 and ADE20K); I would test whether adding more tasks or larger auxiliary data continues to improve the encoder or saturates quickly.
  • The scheme sits between autoregression and inpainting, since masking several consecutive tokens makes the prediction task partially like masked image modeling; I would expect a unified loss to capture both, and the paper's masking-ratio ablations give evidence for that view.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes MAL, a self-supervised pretraining framework for xLSTM-based vision backbones. MAL combines a 'cluster-masked' autoregressive objective, in which spatially adjacent patches are grouped into clusters that serve as prediction units, with a two-stage pretraining procedure: first image autoregression on ImageNet-1K, then multi-task pretraining on depth estimation (NYU Depth v2) and semantic segmentation (ADE20K) jointly with autoregression. The authors report consistent accuracy improvements over supervised baselines and over the ViL xLSTM backbone: MAL-B reaches 83.4% top-1 on ImageNet-1K versus 82.4% for ViL-B, and MAL-S improves ADE20K mIoU to 47.5 versus 46.3 for ViL-S. The paper also reports robustness gains on ImageNet variants and an ablation study covering prediction-unit size, scanning order, masking ratio, decoder design, and multi-task pretraining.

Significance. If the results hold, the paper would be a useful contribution as the first autoregressive pretraining study for xLSTM vision backbones, and the combination of cluster-masked prediction with multi-task encoder-decoder pretraining is a reasonable direction. The paper's strengths include the systematic ablation over prediction-unit granularity and prediction order, the inclusion of out-of-domain ImageNet evaluations, and the segmentation results on ADE20K. However, the significance is substantially weakened by internal inconsistencies in the definition of the pretraining objective and in the ablation tables, and by the absence of a precise algorithmic description or code for the cluster-masked procedure. The central claim that MAL 'sets a new benchmark' therefore cannot be assessed reliably from the manuscript in its current form.

major comments (5)
  1. [§3.2.1, Eq. (3) and §4.5, Table 6] The formal cluster-autoregressive objective and the best-performing configuration are mutually inconsistent. With the chosen 4×4 cluster size on a 192×192 input there are only (192/16/4)^2 = 9 prediction units, but Table 6 says the pretraining sequence is 144 tokens and reports a masking ratio of 20% (28 tokens), which cannot be expressed in units of 9 clusters. The paper never defines how random token masking and cluster prediction units interact, so the mechanism behind the headline 83.4% is not verifiable from the text; please provide the exact mask construction and state explicitly whether the 83.4% result uses the Eq. (3) objective or a length-preserving inpainting objective.
  2. [§3.3.2, Eq. (4)] The causal mask is written as content_mask_ij = 0 for i < j and −∞ for i ≥ j, which by the paper's own definition ('0 allows the i-th token to attend to the j-th token') lets later tokens attend to earlier ones and blocks self/previous attention; this is the opposite of the stated property that 'each token attends only to itself and preceding tokens'. If this is a sign or indexing typo, it must be corrected, because the autoregressive property of the encoder-decoder is load-bearing for the method.
  3. [§4.5, Tables 4, 6, 7 and §4.2, Table 1] The hyperparameters (cluster size, masking ratio, decoder depth and width) are selected by maximizing ImageNet validation top-1 accuracy on the same validation set used to report the headline 83.4%, so the comparison against ViL-B (82.4%) is not a comparison of a fixed architecture but of the best cell of a searched grid. The paper should report an evaluation protocol that separates configuration selection from reporting, or provide multiple seeds with error bars; otherwise the 1.0% claimed gain over ViL-B may be partially due to selection noise.
  4. [§4.5, Tables 4 and 8] The ablation story is internally inconsistent. Table 4 reports 83.4% for 9 cluster units under what the text describes as autoregressive pretraining, while Table 8 reports 82.5% for MAL-B with autoregression only and credits multi-task pretraining for the rise to 83.4%. If Table 4's configuration already includes multi-task pretraining, then the 1.5-point cluster-size gain and the 0.9-point multi-task gain cannot both be read off as stated. Please specify exactly which pretraining stages and which hyperparameters each ablation uses.
  5. [§4.2 and Table 1] The throughput for MAL-T is reported as 4108 imgs/s in Table 1 but as 1301 imgs/s in the text. Since efficiency is one of the paper's stated advantages, this discrepancy must be resolved before the efficiency claims can be assessed.
minor comments (5)
  1. [Table 1 caption] The caption reads 'all image sizes are 2242'; this should be '224×224' or '224^2'.
  2. [§2.2] The section begins with a sentence fragment: 'Unified Architecture has become increasingly popular...' should be a complete sentence with a subject.
  3. [References] References [1] and [2] are the same paper (Vision-LSTM by Alkin et al.) and should be merged.
  4. [§4.5, Table 6] The masking ratios are labeled as 1%, 10%, 20%, 30%, 50%, and 70%, but for a 144-token sequence the corresponding token counts are 1, 14, 28, 43, 72, and 100, which are approximations (e.g., 14/144 = 9.7%); the rounding convention should be stated.
  5. [§3.3.4, Eq. (6)] The decoder applies the same content mask as the encoder, but if the decoder is reconstructing all tokens with position embeddings, the role of the causal mask in the decoder should be explained, especially in light of the length-preserving reconstruction described in Table 6.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the reported gains are empirical training and evaluation outcomes, not derivations that reduce to their inputs.

full rationale

The paper's central results in Tables 1, 2, and 3 are measured accuracies of trained models compared with external published baselines; none of the reported numbers is defined in terms of another reported number, and none of the pretraining objectives in Eqs. (1)-(3) is fitted to the benchmark value it is claimed to predict. The ablations in Tables 4, 6, 7, and 8 do select hyperparameters such as cluster size, masking ratio, and decoder depth/width by ImageNet validation accuracy, and the headline model uses the best configuration found; this is a model-selection and validation-set-reuse concern, not a circular derivation, because no selected hyperparameter is substituted back into the objective in a way that makes the final accuracy equal to the selection criterion by construction. Similarly, the inconsistency between the strict causal mask in Eq. (4) and the multi-token inpainting setup described in Section 4.5 is an internal-consistency and reproducibility concern, not a circular step, since Eq. (4) does not define the reported 83.4% accuracy. There is no load-bearing self-citation chain: the xLSTM, ViL, and ARM references are external works, and the paper invokes no uniqueness theorem from the authors' own prior work. The paper is largely self-contained against external benchmarks, so the honest finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The method is an empirical recipe, not a derivation. The architecture is inherited from prior xLSTM/ViL work, and the core pretraining assumption (autoregressive cluster prediction transfers) is tested only through ablations. The reported accuracy depends on several hyperparameters chosen by validation accuracy on the same benchmark, plus unreported loss weights.

free parameters (5)
  • Cluster size (number of prediction units) = 9 (4x4 clusters)
    Selected in Table 4 ablation on ImageNet-1K validation; yields 83.4% top-1.
  • Masking ratio = 20%
    Selected in Table 6 ablation on ImageNet-1K validation; yields 83.4% top-1.
  • Decoder depth and width = depth 8, width 512
    Selected in Table 7 ablation on ImageNet-1K validation.
  • Multi-task loss weights alpha and beta = not reported
    Weighting in Eq. (9) is not specified; the balance between depth/segmentation and autoregression losses is a free choice.
  • Prediction order = row-first forward
    Adopted in Section 4.5 after ablation Table 5 showed only small differences among fixed orders.
assumptions (4)
  • domain assumption The xLSTM architecture (mLSTM blocks with exponential gating and matrix memory) provides a valid backbone for vision when adapted as in ViL.
    The method builds on the xLSTM/ViL architecture from [4, 2] and does not re-derive or validate its properties.
  • domain assumption Autoregressive prediction of clusters in a fixed raster order learns transferable visual representations.
    This is the core pretraining assumption (Section 3.2); no proof is offered, only empirical support via ablations.
  • domain assumption Adding depth estimation and segmentation pretraining on NYUv2 and ADE20K improves ImageNet classification.
    Stage 2 multi-task pretraining (Section 3.4.2) assumes positive transfer from these tasks to classification.
  • domain assumption The ImageNet, ADE20K, and NYU Depth v2 datasets are correctly labeled and representative.
    Results rely on standard benchmark labels without independent verification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance." pith.science (2026). https://pith.science/paper/JBQCOQ2B

@misc{pith2026241210730,
  author       = {Pith},
  title        = {Pith review of: MAL: Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JBQCOQ2B}},
  note         = {Machine review of arXiv:2412.10730}
}
read the original abstract

The Long Short-Term Memory (LSTM) networks have traditionally faced challenges in scaling and effectively capturing complex dependencies in visual tasks. The xLSTM architecture has emerged to address these limitations, incorporating exponential gating and a parallel matrix memory structure to enhance performance and scalability. Despite these advancements, the potential of xLSTM in visual computing has not been fully realized, particularly in leveraging autoregressive techniques for improved feature extraction. In this paper, we introduce MAL (Cluster-Masked and Multi-Task Pretraining for Enhanced xLSTM Vision Performance), a novel framework that enhances xLSTM's capabilities through innovative pretraining strategies. We propose a cluster-masked masking method that significantly improves local feature capture and optimizes image scanning efficiency. Additionally, our universal encoder-decoder pretraining approach integrates multiple tasks, including image autoregression, depth estimation, and image segmentation, thereby enhancing the model's adaptability and robustness across diverse visual tasks. Our experimental results demonstrate that MAL surpasses traditional supervised models and fully leverages the scaling potential of xLSTM, setting a new benchmark in visual task performance.

Figures

Figures reproduced from arXiv: 2412.10730 by the authors.

Figure 1
Figure 1. Overall architecture. (a) Input Image (b) Pixel-based (c) Pixel-masked (d) Patch-based (e) Cluster-based (f) Cluster-masked(ours) [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Different prediction units in the autoregressive modeling. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Different prediction orderings of a visual sentence. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: pretrain. in a sequence, capturing intricate spatial relationships and building a strong foundation in image structures. 3.4.2 Stage 2: Multi-Task Pretraining The second stage builds on the autoregressive capabilities from Stage 1 through multi-task pretraining (see Fi…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 28 canonical work pages

  1. [1]

    Alkin, M

    B. Alkin, M. Beck, K. Pöppel, S. Hochreiter, and J. Brandstetter. Vision-lstm: xlstm as generic vision backbone. arXiv:2406.04303, 2024

    arXiv 2024

  2. [2]

    Alkin, M

    B. Alkin, M. Beck, K. Pöppel, S. Hochreiter, and J. Brandstetter. Vision-lstm: xlstm as generic vision backbone, 2024

    work page 2024

  3. [3]

    H. Bao, L. Dong, S. Piao, and F. Wei. BEiT: BERT pre-training of image transformers. In International Conference on Learning Representations, 2022

    work page 2022

  4. [4]

    M. Beck, K. Pöppel, M. Spanring, A. Auer, O. Prudnikova, M. Kopp, G. Klambauer, J. Brandstetter, and S. Hochreiter. xlstm: Extended long short-term memory. arXiv:2405.04517, 2024

    arXiv 2024

  5. [5]

    Beyer, O

    L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. v. d. Oord. Are we done with imagenet? arXiv preprint arXiv:2006.07159, 2020

    arXiv 2006

  6. [6]

    M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever. Generative pretraining from pixels. In ICML, 2020

    work page 2020

  7. [7]

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations. In ICLR, 2020

    work page 2020

  8. [8]

    T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. Hinton. Pix2seq: A language modeling framework for object detection, 2022

    work page 2022

Show all 54 references
  1. [9]

    X. Chen, H. Fan, R. Girshick, and K. He. Improved baselines with momentum contrastive learning. preprint arXiv:2003.04297, 2020

    2003 arXiv

  2. [10]

    X. Chen, S. Xie, and K. He. An empirical study of training self-supervised vision transformers. ArXiv, abs/2104.02057, 2021

    2021 arXiv

  3. [11]

    Cheng, I

    B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation, 2022. 13

    2022

  4. [12]

    S. De, S. L. Smith, A. Fernando, A. Botev, G. Cristian-Muraru, A. Gu, R. Haroun, L. Berrada, Y . Chen, S. Srinivasan, G. Desjardins, A. Doucet, D. Budden, Y . W. Teh, R. Pascanu, N. D. Freitas, and C. Gul- cehre. Griffin: Mixing gated linear recurrences with local attention fo...

    2024 arXiv

  5. [13]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248–255. Ieee, 2009

    2009

  6. [14]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009

    2009

  7. [15]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2020

    2020

  8. [16]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    2021

  9. [18]

    Y . Duan, W. Wang, Z. Chen, X. Zhu, L. Lu, T. Lu, Y . Qiao, H. Li, J. Dai, and W. Wang. Vision-rwkv: Efficient and scalable visual perception with rwkv-like architectures. CoRR, abs/2403.02308, 2024

    2024 arXiv

  10. [19]

    El-Nouby, M

    A. El-Nouby, M. Klein, S. Zhai, M. A. Bautista, A. Toshev, V . Shankar, J. M. Susskind, and A. Joulin. Scalable pre-training of large autoregressive image models. arXiv preprint arXiv:2401.08541, 2024

    2024 arXiv

  11. [20]

    Gu and T

    A. Gu and T. Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023

    2023

  12. [21]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022

    2022

  13. [22]

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020

    2020

  14. [23]

    Hendrycks, S

    D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021

    2021

  15. [24]

    Hendrycks, K

    D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. CVPR, 2021

    2021

  16. [25]

    Hu and A

    R. Hu and A. Singh. Unit: Multimodal multitask learning with a unified transformer, 2021

    2021

  17. [26]

    T. Hua, Y . Tian, S. Ren, M. Raptis, H. Zhao, and L. Sigal. Self-supervision through random segments with autoregressive coding (randsac). In ICLR, 2022

    2022

  18. [27]

    Izmailov, D

    P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407, 2018

    2018 arXiv

  19. [28]

    J. Jain, J. Li, M. Chiu, A. Hassani, N. Orlov, and H. Shi. Oneformer: One transformer to rule universal image segmentation, 2022

    2022

  20. [29]

    Kolesnikov, A

    A. Kolesnikov, A. S. Pinto, L. Beyer, X. Zhai, J. Harmsen, and N. Houlsby. Uvim: A unified modeling approach for vision with learned guiding codes, 2022

    2022

  21. [30]

    Liang, X

    D. Liang, X. Zhou, X. Wang, X. Zhu, W. Xu, Z. Zou, X. Ye, and X. Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv:2402.10739, 2024

    2024 arXiv

  22. [31]

    Z. Liu, H. Mao, C. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, pages 11966–11976. IEEE, 2022

    2022

  23. [32]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019

    2019

  24. [33]

    J. Lu, C. Clark, R. Zellers, R. Mottaghi, and A. Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks, 2022. 14

    2022

  25. [34]

    P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rgbd images. In ECCV, 2012

    2012

  26. [35]

    Orvieto, S

    A. Orvieto, S. L. Smith, A. Gu, A. Fernando, C. Gulcehre, R. Pascanu, and S. De. Resurrecting recurrent neural networks for long sequences. In ICML, 2023

    2023

  27. [36]

    B. Peng, E. Alcaide, Q. G. Anthony, A. Albalak, S. Arcadinho, S. Biderman, H. Cao, X. Cheng, M. N. Chung, L. Derczynski, et al. Rwkv: Reinventing rnns for the transformer era. 2023

    2023

  28. [37]

    B. Peng, D. Goldstein, Q. Anthony, A. Albalak, E. Alcaide, S. Biderman, E. Cheah, X. Du, T. Ferdinan, H. Hou, P. Kazienko, K. K. GV , J. Koco´n, B. Koptyra, S. Krishna, R. M. J. au2, N. Muennighoff, F. Obeid, A. Saito, G. Song, H. Tu, S. Wo´ zniak, R. Zhang, B. Zhao, Q. Zhao, ...

    2024 arXiv

  29. [38]

    Y . Qi, F. Yang, Y . Zhu, Y . Liu, L. Wu, R. Zhao, and W. Li. Exploring stochastic autoregressive image modeling for visual representation. In AAAI, 2023

    2023

  30. [39]

    Z. Qin, S. Yang, and Y . Zhong. Hierarchically gated recurrent neural network for sequence modeling. In NeurIPS, 2023

    2023

  31. [40]

    S. Ren, X. Li, H. Tu, F. Wang, F. Shu, L. Zhang, J. Mei, L. Yang, P. Wang, H. Wang, A. Yuille, and C. Xie. Autoregressive pretraining with mamba in vision, 2024

    2024

  32. [41]

    S. Ren, Z. Wang, H. Zhu, J. Xiao, A. Yuille, and C. Xie. Rejuvenating image-gpt as strong visual representation learners. arXiv preprint arXiv:2312.02147, 2023

    2023 arXiv

  33. [42]

    S. Ren, F. Wei, Z. Zhang, and H. Hu. Tinymim: An empirical study of distilling mim pre-trained models. In CVPR, 2023

    2023

  34. [43]

    Y . Sun, L. Dong, S. Huang, S. Ma, Y . Xia, J. Xue, J. Wang, and F. Wei. Retentive network: A successor to transformer for large language models. arXiv:2307.08621, 2023

    2023 arXiv

  35. [44]

    Y . Sun, X. Li, K. Dalal, J. Xu, A. Vikram, G. Zhang, Y . Dubois, X. Chen, X. Wang, S. Koyejo, et al. Learning to (learn at test time): Rnns with expressive hidden states. arXiv:2407.04620, 2024

    2024 arXiv

  36. [45]

    Touvron, M

    H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML, volume 139 of Proceedings of Machine Learning Research, pages 10347–10357. PMLR, 2021

    2021

  37. [46]

    Touvron, M

    H. Touvron, M. Cord, A. El-Nouby, J. Verbeek, and H. Jégou. Three things everyone should know about vision transformers. In ECCV (24), volume 13684 of Lecture Notes in Computer Science, pages 497–515. Springer, 2022

    2022

  38. [47]

    Touvron, M

    H. Touvron, M. Cord, and H. Jégou. Deit III: revenge of the vit. In ECCV (24), volume 13684 of Lecture Notes in Computer Science, pages 516–533. Springer, 2022

    2022

  39. [48]

    F. Wang, J. Wang, S. Ren, G. Wei, J. Mei, W. Shao, Y . Zhou, A. Yuille, and C. Xie. Mamba-r: Vision mamba also needs registers. arXiv preprint arXiv:2405.14858, 2024

    2024 arXiv

  40. [49]

    H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In NeurIPS, 2019

    2019

  41. [50]

    T. Xiao, Y . Liu, B. Zhou, Y . Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In V . Ferrari, M. Hebert, C. Sminchisescu, and Y . Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings,...

    2018

  42. [51]

    S. Yang, B. Wang, Y . Shen, R. Panda, and Y . Kim. Gated linear attention transformers with hardware- efficient training. arXiv:2312.06635, 2023

    2023 arXiv

  43. [52]

    S. Zhai, N. Jaitly, J. Ramapuram, D. Busbridge, T. Likhomanenko, J. Y . Cheng, W. Talbott, C. Huang, H. Goh, and J. Susskind. Position prediction as an effective pretraining strategy. arXiv preprint arXiv:2207.07611, 2022

    2022 arXiv

  44. [53]

    B. Zhou, H. Zhao, F. X. P. Fernandez, S. Fidler, and A. Torralba. Scene parsing through ade20k dataset. IEEE, 2017

    2017

  45. [54]

    B. Zhou, H. Zhao, X. Puig, T. Xiao, S. Fidler, A. Barriuso, and A. Torralba. Semantic understanding of scenes through the ADE20K dataset. Int. J. Comput. Vis., 127(3):302–321, 2019

    2019

  46. [55]

    L. Zhu, B. Liao, Q. Zhang, X. Wang, W. Liu, and X. Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 15

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.