Pith. sign in

REVIEW 4 major objections 5 minor 105 references

OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read OmniVec2 proposes a single shared transformer with modality-specific tokenizers and task heads, claiming state-of-the-art or competitive results on 25 datasets across 12 modalities via pairwise cross-attention training.

desk verdict OmniVec2 is a plausible generalist multimodal architecture, but its headline SOTA claims rest on an unmatched evaluation protocol and inconsistent numbers, so treat the empirical results as unverified. read the letter →

arxiv 2507.13364 v1 pith:5BSEDU3S submitted 2025-07-06 cs.CV cs.AI

Siddharth Srivastava , Gaurav Sharma This is my paper
classification cs.CVcs.AI
keywords multimodallearningmultitasktransformermaskedpretrainingcross-attentionfusionmodality-specifictokenizerssharedbackbonegeneralistmodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

OmniVec2 is a neural architecture and training recipe for learning one model that handles many data types at once. The paper's central claim is that a shared transformer backbone, fronted by separate tokenizers for each modality and topped with task-specific heads, can match or beat both specialized single-modality models and earlier generalist models on 25 benchmark datasets spanning image, video, audio, text, point clouds, graphs, tables, time series, X-ray, infrared, hyperspectral, and IMU data. The training scheme alternates between pairs of modalities during masked pretraining and during supervised multitask training, so the network learns to fuse and share information across modalities without requiring paired data. If the claim holds, a single network could replace a collection of modality-specific models and transfer to new datasets and even new modalities with only lightweight adaptation.

What carries the argument

The mechanism that carries the argument is the two-stage pairwise fusion loop. Stage 1 masked pretraining trains the shared transformer f on each modality alone; Stage 2 masked pretraining passes pairs of modalities through f, fuses them with cross-attention, passes the fused features through the second shared transformer g, and reconstructs masked tokens for both modalities; supervised training does the same with task heads instead of decoders. This iterated modality-pair training is what the paper identifies as enabling cross-modal knowledge sharing and reliable, regularized representations.

What would settle it

Re-run the paper's exact unseen-dataset protocol (freeze base embeddings, train two fully connected layers on 10% of the training set) for the cited baselines using their released code and the same 10% split; if those baselines match or exceed OmniVec2's numbers, the central claim that the shared backbone transfers better than specialized and generalist models is refuted.

Watch

Extended reading notes

Core claim

OmniVec2's central discovery is that cross-modal knowledge sharing can be achieved with a common transformer bottleneck rather than separate per-modality encoders. Each modality is tokenized by its own tokenizer; the token streams pass through a shared transformer f, are fused with another stream via a cross-attention module, pass through a second shared transformer g, and are fused back with the original modality features before task-specific heads produce predictions. At inference the fusion branches are dropped, giving hmt ∘ g ∘ f(x). The training alternates sampling pairs of modalities and tasks, so the sum of losses in the multimodal multitask objective is stochastically minimized two losses at a time. The paper reports top-1 accuracy of 94.6% on iNaturalist-2018, 93.6% on Kinetics-400, 99.1% on ESC50, 90.6% overall accuracy on Indian Pines, and lower error than prior art on ModelNet40-C, among other results.

Load-bearing premise

The paper's claim that OmniVec2 beats earlier methods on unseen datasets rests on comparing its frozen-embedding, 10%-data results with published benchmark numbers that may come from full fine-tuning; if those numbers are not directly comparable, the claim of superiority is not established.

Editorial extensions

If this is right

  • If the results hold, one shared transformer can replace modality-specific encoder stacks across vision, audio, text, point cloud, and other domains, shrinking model size and compute.
  • The ablations quantify that multimodal pairwise pretraining beats unimodal pretraining by roughly 16% on iNaturalist-2018 and Kinetics-400 and by 8% on ESC50, meaning cross-modal pretraining is a major source of the gains.
  • Modality-specific tokenizers outperform a unified tokenizer by about 4-5% average, and task-specific heads outperform a unified head by about 3.5% average, in the paper's comparisons.
  • Because random modality pairing works with task balancing, the method avoids hand-picking which modality pairs to co-train, simplifying the training pipeline.
  • Frozen embeddings with two fully connected layers transfer to unseen datasets and unseen modalities, suggesting the learned representation is broadly reusable.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • One consequence the paper leaves implicit: if the frozen-embedding protocol transfers to modalities as different as IMU and graphs, then a modest set of pretraining domains might suffice for a much wider range of sensor data, making new-modality adaptation a linear-probe problem.
  • Because inference drops the cross-attention branches entirely, the architecture at test time is a plain single-stream transformer; this suggests the reported gains could be a pretraining effect rather than a fusion effect. A clean test would be to train the same single-stream model with the same two-stage pairwise pretraining but without cross-attention, to separate the contributions.
  • The pairwise sampling scheme could be made adaptive: instead of uniform random pairs, one could sample modality pairs by gradient conflict or task difficulty, a direction the paper does not explore but which its task-balancing idea naturally extends to.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. OmniVec2 is a transformer-based multimodal multitask model with modality-specific tokenizers, a shared BERT-style transformer backbone, cross-attention based pairwise feature fusion during training, and task-specific ViT-Tiny heads. Training proceeds in three stages: unimodal masked autoencoding, pairwise multimodal masked autoencoding, and pairwise multitask supervised training with task balancing. The paper evaluates the method on 25 datasets spanning image, video, audio, text, point cloud, depth, infrared, hyperspectral, X-ray, IMU, graph, time-series, and tabular modalities, reporting state-of-the-art or near-state-of-the-art results against specialized baselines and generalist models such as OmniVec, Meta-Transformer, and Uni-Perceiver. The paper also includes ablations on tokenizer type, task head type, and pretraining scheme.

Significance. If the empirical claims were supported, the paper would contribute a useful generalist architecture that combines modality-specific tokenizers with a shared transformer backbone and pairwise multimodal training, an approach that could lower the cost of extending unified models to new modalities. The paper has real strengths: it covers an unusually wide range of modalities, includes an ablation that isolates the effect of modality-specific tokenizers and task-specific heads, and compares against both specialized and generalist baselines. However, the load-bearing state-of-the-art claim is not currently supported: the unseen-dataset protocol is not comparable to the published baselines, several key numbers conflict internally, and no error bars or reproducibility artifacts are provided. With corrected evaluation and reporting, the architecture idea may be publishable, but the present manuscript requires substantial revision.

major comments (4)
  1. [Section 4.2 and Table 8] The evaluation protocol for unseen datasets is not comparable to the published SOTA numbers. The text states that the authors freeze the base embeddings, do not fine-tune the full network, use a simpler task head, and train only two fully connected layers on 10% of each training set, while the reported SOTA values, mostly from OmniVec, come from full-data fine-tuning protocols. The reported gains, such as +0.4 on Oxford-IIIT Pets, +1.1 on ScanObjectNN, and +2.8 on NYUv2, cannot be attributed to the architecture under this protocol mismatch. Either re-run the baselines under the same 10%-frozen-backbone protocol, or present these results as linear-probing transfer results and remove the SOTA comparison. This point is load-bearing because the central claim of state-of-the-art generalization rests on this table.
  2. [Section 4.1, Section 4.3, and Table 8] Several internally inconsistent numbers make the claimed results ambiguous. The image paragraph reports 89.3% accuracy on ImageNet, while the later paragraph on pretraining datasets reports 93.6% top-1 accuracy on ImageNet-1K. Table 8 lists MSR-VTT Recall@10 as 85.8, but Section 4.3 says the fine-tuned method achieves 89.4%. In addition, Section 4.4 refers to the infrared dataset as RegDB while Table 10a identifies it as SYSU-MM01. These contradictions must be reconciled before the reported state-of-the-art numbers can be trusted.
  3. [Section 4.5 and Table 12] The ablation interpretation is numerically inconsistent with the table. The text says that multimodal multitask pretraining using the proposed approach, called row 5, outperforms unimodal pretraining, row 1, by about 16% on iNaturalist and Kinetics-400 and by about 8% on ESC50. Row 5 versus row 1 actually gives 86.1-74.2=11.9, 85.2-78.6=6.6, and 87.0-82.4=4.6. The claimed 16% matches row 6 only for iNaturalist, and even then the Kinetics-400 and ESC50 deltas are much smaller. Since row 6, not row 5, is the full proposed configuration, the ablation should be re-reported and re-interpreted.
  4. [All benchmark tables] No error bars, confidence intervals, or repeated-run statistics are reported for any of the benchmark results. Several reported gaps are smaller than half a percentage point, for example iNaturalist 94.6 versus 93.8, ESC50 99.1 versus 98.4, and YouCook2 69.9 versus 70.8 for the fine-tuned baseline. Without variance estimates or at least multiple seeds, the state-of-the-art superiority cannot be assessed statistically. The authors should report mean and standard deviation over multiple runs or otherwise justify that the differences are not noise.
minor comments (5)
  1. [Abstract] The abstract says the method handles approximately 12 modalities, but the enumerated list contains 13 entries: image, video, audio, text, depth, point cloud, time series, tabular, graph, X-ray, infrared, IMU, and hyperspectral.
  2. [Throughout] The manuscript contains multiple typos and grammatical slips, including modalies, traning, lets up stochastically, and the phrase denoted ahmt in Figure 1. A careful copy-edit is needed.
  3. [Section 3 and Section 4] Important implementation details are deferred to the supplementary material, including the exact tokenizers per modality, the task-balancing formula, hyperparameters, and compute budgets. The supplementary material must be available and complete for the empirical claims to be reproducible.
  4. [Table 12] The column configuration in Table 12 is confusing: the Modality column and Tokenizer column are merged into a single token such as Multiple Unified or Multiple Modality, making it hard to map rows to the ablation text. Please separate the factors and label each row explicitly.
  5. [Table 8] The column header 3-Fold Accuracy for UCF-101 and HMDB51 is nonstandard; these datasets typically use three splits, and the header should say 3-Split Accuracy or similar.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the architecture, pretraining strategy, and benchmark evaluations are self-contained; Section 4.2 raises a comparability concern but not a circular derivation.

full rationale

This is an empirical systems paper rather than a derivation, and none of the load-bearing claims reduce to their own inputs. The optimization objective in Eq. 1 is a standard multitask loss, and the network is presented as a proposed architecture rather than derived from that objective, so there is no self-definitional step. No fitted parameter is renamed as a prediction: the pretraining and task-training procedures are described independently of the reported accuracies, and the SOTA comparisons are external, falsifiable benchmark numbers. The paper does cite the authors' own prior OmniVec [67] as a motivation and as a baseline, but that citation is not load-bearing: the proposed shared-backbone design is not justified by OmniVec's results, and the comparisons to OmniVec are ordinary benchmark claims rather than a mathematical reduction to prior work. The most substantive concern is in Section 4.2, where the paper states that for unseen datasets it freezes base embeddings and trains only two fully connected layers on 10% of each training set, unlike other methods that fine-tune the full network; comparing those numbers with published full-fine-tuning results could make the claimed margins non-apples-to-apples. That is an experimental-protocol limitation affecting correctness or fairness of comparison, not circularity, because the reported numbers are not constructed from the baseline numbers and the claim remains empirically testable. Accordingly, the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

No new physical or mathematical entities are postulated; the contribution is an architecture and training recipe. The load-bearing unstated assumptions are evaluation comparability and transfer from pairwise to joint training.

free parameters (4)
  • Mask ratios per modality = 95% image/video/audio, 90% point cloud, 95% text
    Hand-chosen pretraining hyperparameters; no sensitivity analysis is reported, so their effect on downstream SOTA is unknown.
  • Pretraining epochs = 3000
    Fixed without reported ablation; expensive and likely material to representation quality.
  • Token prediction fraction f = 5%
    Chosen following prior work; no ablation for OmniVec2's multi-modal setting.
  • Task balancing weights = not specified in main text
    Loss weighting based on convergence rate is described only qualitatively; the exact schedule or coefficients are not in the main text.
assumptions (3)
  • domain assumption Pairs of randomly sampled modalities during training are sufficient for cross-modal knowledge sharing across all modalities.
    The paper provides no proof and only a limited ablation; the entire pairwise training strategy rests on this.
  • domain assumption The frozen-embedding linear probe on 10% of unseen dataset training data is a valid comparison to published SOTA numbers.
    Sections 4.2 and 4.4 compare against specialized baselines without matching training protocols.
  • domain assumption Modality-specific tokenizers for all 12 modalities can be built with sufficient fidelity using the unspecified recipes in the supplementary.
    Tokenization is described as similar to Uni-Perceiver with details deferred; the core architecture depends on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning." pith.science (2026). https://pith.science/paper/5BSEDU3S

@misc{pith2026250713364,
  author       = {Pith},
  title        = {Pith review of: OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5BSEDU3S}},
  note         = {Machine review of arXiv:2507.13364}
}
read the original abstract

We present a novel multimodal multitask network and associated training algorithm. The method is capable of ingesting data from approximately 12 different modalities namely image, video, audio, text, depth, point cloud, time series, tabular, graph, X-ray, infrared, IMU, and hyperspectral. The proposed approach utilizes modality specialized tokenizers, a shared transformer architecture, and cross-attention mechanisms to project the data from different modalities into a unified embedding space. It addresses multimodal and multitask scenarios by incorporating modality-specific task heads for different tasks in respective modalities. We propose a novel pretraining strategy with iterative modality switching to initialize the network, and a training algorithm which trades off fully joint training over all modalities, with training on pairs of modalities at a time. We provide comprehensive evaluation across 25 datasets from 12 modalities and show state of the art performances, demonstrating the effectiveness of the proposed architecture, pretraining strategy and adapted multitask training.

Figures

Figures reproduced from arXiv: 2507.13364 by the authors.

Figure 1
Figure 1. Overview of the proposed method. The proposed method consists of three parts, the feature transformation network f(·) which consists of a transformer followed by fully connected layers to reduce feature dimensions, another transformer g(·) and finally the task prediction heads hmt(·) for task t on modality m. The input data is tokenized with corresponding modality specific tokenizer. While training, pairs of modalit… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

105 extracted references · 47 canonical work pages

  1. [1]

    Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text

    Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34:24206–24221, 2021. 1, 2, 4, 5, 6

    2021

  2. [2]

    Objects that sound

    Relja Arandjelovic and Andrew Zisserman. Objects that sound. In Proceedings of the European conference on com- puter vision (ECCV), pages 435–451, 2018. 2

    2018

  3. [3]

    3d seman- tic parsing of large-scale indoor spaces

    Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioan- nis Brilakis, Martin Fischer, and Silvio Savarese. 3d seman- tic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recog- nition, pages 1534–1543, 2016. 5

    2016

  4. [4]

    Mae- ast: Masked autoencoding audio spectrogram transformer

    Alan Baade, Puyuan Peng, and David Harwath. Mae- ast: Masked autoencoding audio spectrogram transformer. arXiv preprint arXiv:2203.16691, 2022. 2

    arXiv 2022

  5. [5]

    Data2vec: A general frame- work for self-supervised learning in speech, vision and lan- guage

    Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. Data2vec: A general frame- work for self-supervised learning in speech, vision and lan- guage. In International Conference on Machine Learning, pages 1298–1312. PMLR, 2022. 1, 2

    2022

  6. [6]

    Generative adversarial networks based on transformer encoder and convolution block for hyperspectral image classification

    Jing Bai, Jiawei Lu, Zhu Xiao, Zheng Chen, and Licheng Jiao. Generative adversarial networks based on transformer encoder and convolution block for hyperspectral image classification. Remote Sensing, 14(14):3426, 2022. 8

    2022

  7. [7]

    Hierarchical perceiver

    Joao Carreira, Skanda Koppula, Daniel Zoran, Adria Re- casens, Catalin Ionescu, Olivier Henaff, Evan Shelhamer, Relja Arandjelovic, Matt Botvinick, Oriol Vinyals, et al. Hierarchical perceiver. arXiv preprint arXiv:2202.10890 ,

    arXiv

  8. [8]

    Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection

    Ke Chen, Xingjian Du, Bilei Zhu, Zejun Ma, Taylor Berg- Kirkpatrick, and Shlomo Dubnov. Hts-at: A hierarchical token-semantic audio transformer for sound classification and detection. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 646–650. IEEE, 2022. 6

    2022

Show all 105 references
  1. [9]

    Dialogsum: A real-life scenario dialogue summarization dataset

    Yulong Chen, Yang Liu, Liang Chen, and Yue Zhang. Dialogsum: A real-life scenario dialogue summarization dataset. arXiv preprint arXiv:2105.06762, 2021. 5

    2021 arXiv

  2. [10]

    Multi-task learning with deep neu- ral networks: A survey

    Michael Crawshaw. Multi-task learning with deep neu- ral networks: A survey. arXiv preprint arXiv:2009.09796,

    2009 arXiv

  3. [11]

    One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code

    Yong Dai, Duyu Tang, Liangxin Liu, Minghuan Tan, Cong Zhou, Jingquan Wang, Zhangyin Feng, Fan Zhang, Xueyu Hu, and Shuming Shi. One model, multiple modalities: A sparsely activated approach for text, sound, image, video and code. arXiv preprint arXiv:2205.06126, 2022. 2

    2022 arXiv

  4. [12]

    Imagenet: A large-scale hierarchical im- age database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical im- age database. In CVPR, pages 248–255. Ieee, 2009. 5, 6

    2009

  5. [13]

    BERT: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional trans- formers for language understanding. InNAACL-HLT, 2019. 3, 5, 6, 7

    2019

  6. [14]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv...

    2010 arXiv

  7. [15]

    A generaliza- tion of transformer networks to graphs

    Vijay Prakash Dwivedi and Xavier Bresson. A generaliza- tion of transformer networks to graphs. AAAI Workshop on Deep Learning on Graphs: Methods and Applications ,

  8. [16]

    Efficiently identifying task group- ings for multi-task learning

    Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task group- ings for multi-task learning. Advances in Neural Informa- tion Processing Systems, 34:27503–27516, 2021. 4

    2021

  9. [17]

    End-to-end audio strikes back: Boosting augmentations towards an efficient audio classification net- work

    Avi Gazneli, Gadi Zimerman, Tal Ridnik, Gilad Sharir, and Asaf Noy. End-to-end audio strikes back: Boosting augmentations towards an efficient audio classification net- work. arXiv preprint arXiv:2204.11479, 2022. 6

    2022 arXiv

  10. [18]

    Audio set: An ontology and human- labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- labeled dataset for audio events. In 2017 IEEE interna- tional conference on acoustics, speech and signal process- ...

    2017

  11. [19]

    Omnimae: Single model masked pretraining on images and videos

    Rohit Girdhar, Alaaeldin El-Nouby, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Omnimae: Single model masked pretraining on images and videos. arXiv preprint arXiv:2206.08356, 2022. 2, 4, 6

    2022 arXiv

  12. [20]

    Omni- vore: A single model for many visual modalities

    Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omni- vore: A single model for many visual modalities. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16102–16112, 2022. 1, 2, 6

    2022

  13. [21]

    Samsum corpus: A human-annotated dia- logue dataset for abstractive summarization

    Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Alek- sander Wawer. Samsum corpus: A human-annotated dia- logue dataset for abstractive summarization. arXiv preprint arXiv:1911.12237, 2019. 5

    1911 arXiv

  14. [22]

    Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778,

    Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer.arXiv preprint arXiv:2104.01778,

    arXiv

  15. [23]

    Uavm: Towards unifying audio and visual models

    Yuan Gong, Alexander H Liu, Andrew Rouditchenko, and James Glass. Uavm: Towards unifying audio and visual models. IEEE Signal Processing Letters , 29:2437–2441,

  16. [24]

    The” something something” video database for learning and evaluating visual common sense

    Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michal- ski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The” something something” video database for learning and evaluating visual common sense. In ...

    2017

  17. [25]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vi- sio...

    2022

  18. [26]

    Dynamic task prioritization for multitask learning

    Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In Proceedings of the European conference on computer vision (ECCV), pages 270–287, 2018. 2

    2018

  19. [27]

    Maskvit: Masked vi- sual pre-training for video prediction

    Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martin, and Li Fei-Fei. Maskvit: Masked vi- sual pre-training for video prediction. arXiv preprint arXiv:2206.11894, 2022. 2

    2022 arXiv

  20. [28]

    Masked autoencoders are scal- able vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollar, and Ross Girshick. Masked autoencoders are scal- able vision learners. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 16000–16009, 2022. 6

    2022

  21. [29]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. 3

    2016 arXiv

  22. [30]

    Spectral- former: Rethinking hyperspectral image classification with transformers

    Danfeng Hong, Zhu Han, Jing Yao, Lianru Gao, Bing Zhang, Antonio Plaza, and Jocelyn Chanussot. Spectral- former: Rethinking hyperspectral image classification with transformers. IEEE Transactions on Geoscience and Re- mote Sensing, 60:1–15, 2021. 8

    2021

  23. [31]

    Unit: Multimodal multitask learning with a unified transformer

    Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 1439–1449, 2021. 2

    2021

  24. [32]

    Ogb-lsc: A large-scale challenge for machine learning on graphs

    Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-lsc: A large-scale challenge for machine learning on graphs. arXiv preprint arXiv:2103.09430, 2021. 5

    2021 arXiv

  25. [33]

    Perceiver io: A general architec- ture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architec- ture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021. 1, 2

    2021 arXiv

  26. [34]

    Perceiver: General perception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In International confer- ence on machine learning, pages 4651–4664. PMLR, 2021. 1, 2

    2021

  27. [35]

    A review of multimodal image matching: Methods and applications

    Xingyu Jiang, Jiayi Ma, Guobao Xiao, Zhenfeng Shao, and Xiaojie Guo. A review of multimodal image matching: Methods and applications. Information Fusion, 73:22–71,

  28. [36]

    One model to learn them all

    Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszko- reit. One model to learn them all. arXiv preprint arXiv:1706.05137, 2017. 2

    2017 arXiv

  29. [37]

    The kinetics human action video dataset

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Vi- ola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017. 5

    2017 arXiv

  30. [38]

    Mind the gap! injecting commonsense knowledge for abstractive dialogue summarization

    Seungone Kim, Se June Joo, Hyungjoo Chae, Chaehyeong Kim, Seung-won Hwang, and Jinyoung Yeo. Mind the gap! injecting commonsense knowledge for abstractive dialogue summarization. arXiv preprint arXiv:2209.00930, 2022. 6

    2022 arXiv

  31. [39]

    Re- former: The efficient transformer

    Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Re- former: The efficient transformer. In ICLR, 2020. 8

    2020

  32. [40]

    Hmdb: a large video database for human motion recognition

    Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 2011 Interna- tional conference on computer vision , pages 2556–2563. IEEE, 2011. 5

    2011

  33. [41]

    Modeling long-and short-term temporal patterns with deep neural networks

    Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanx- iao Liu. Modeling long-and short-term temporal patterns with deep neural networks. In The 41st international ACM SIGIR conference on research & development in informa- tion retrieval, pages 95–104, 2018. 5

    2018

  34. [42]

    Stratified trans- former for 3d point cloud segmentation

    Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified trans- former for 3d point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8500–8509, 2022. 6

    2022

  35. [43]

    Regu- larization strategy for point cloud via rigidly mixed sample

    Dogyoon Lee, Jaeha Lee, Junhyeop Lee, Hyeongmin Lee, Minhyeok Lee, Sungmin Woo, and Sangyoun Lee. Regu- larization strategy for point cloud via rigidly mixed sample. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15900–15909, 2021. 6

    2021

  36. [44]

    Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks

    Hao Li, Jinguo Zhu, Xiaohu Jiang, Xizhou Zhu, Hongsheng Li, Chun Yuan, Xiaohua Wang, Yu Qiao, Xiaogang Wang, Wenhai Wang, et al. Uni-perceiver v2: A generalist model for large-scale vision and vision-language tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Visio...

    2023

  37. [45]

    Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Spatiotemporal learning by arming image vits with video uniformer. arXiv preprint arXiv:2211.09552, 2022. 6

    2022 arXiv

  38. [46]

    Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting

    Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting. In NeurIPS, 2019. 8

    2019

  39. [47]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Ye- ung, and James Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. arXiv preprint arXiv:2203.02053, 2022. 6

    2022 arXiv

  40. [48]

    Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6

    2014

  41. [49]

    Opt: Omni-perception pre-trainer for cross-modal understanding and generation

    Jing Liu, Xinxin Zhu, Fei Liu, Longteng Guo, Zijia Zhao, Mingzhen Sun, Weining Wang, Hanqing Lu, Shiyu Zhou, Jiajun Zhang, et al. Opt: Omni-perception pre-trainer for cross-modal understanding and generation. arXiv preprint arXiv:2107.00249, 2021. 2

    2021 arXiv

  42. [50]

    Pyraformer: Low- complexity pyramidal attention for long-range time series modeling and forecasting

    Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar. Pyraformer: Low- complexity pyramidal attention for long-range time series modeling and forecasting. In ICLR, 2021. 8

    2021

  43. [51]

    Roberta: A ro- bustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A ro- bustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019. 6, 7

    1907 arXiv

  44. [52]

    Moments in time dataset: one million videos for event understanding

    Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ra- makrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl V ondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine in...

    2019

  45. [53]

    Person recognition system based on a combination of body images from visible light and thermal cameras

    Dat Tien Nguyen, Hyung Gil Hong, Ki Wan Kim, and Kang Ryoung Park. Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors, 17(3):605, 2017. 5, 7

    2017

  46. [54]

    N-beats: Neural basis expansion analy- sis for interpretable time series forecasting

    Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio. N-beats: Neural basis expansion analy- sis for interpretable time series forecasting. arXiv preprint arXiv:1905.10437, 2019. 6

    1905 arXiv

  47. [55]

    Cats and dogs

    Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012. 5

    2012

  48. [56]

    Esc: Dataset for environmental sound clas- sification

    Karol J Piczak. Esc: Dataset for environmental sound clas- sification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015–1018, 2015. 5

    2015

  49. [57]

    Re- thinking video vits: Sparse video tubes for joint image and video learning

    AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Re- thinking video vits: Sparse video tubes for joint image and video learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition , pages 2214–2224, 2023. 6

    2023

  50. [58]

    Omninet: A unified architecture for multi-modal multi-task learning

    Subhojeet Pramanik, Priyanka Agrawal, and Aman Hus- sain. Omninet: A unified architecture for multi-modal multi-task learning. arXiv preprint arXiv:1907.07804 ,

    1907 arXiv

  51. [59]

    Point- net++: Deep hierarchical feature learning on point sets in a metric space

    Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Point- net++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS, 2017. 6

    2017

  52. [60]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 7

    2018

  53. [61]

    Reliable tuberculosis de- tection using chest x-ray with deep learning, segmentation and visualization

    Tawsifur Rahman, Amith Khandakar, Muhammad Abdul Kadir, Khandaker Rejaul Islam, Khandakar F Islam, Rashid Mazhar, Tahir Hamid, Mohammad Tariqul Islam, Saad Kashem, Zaid Bin Mahbub, et al. Reliable tuberculosis de- tection using chest x-ray with deep learning, segmentation and ...

    2020

  54. [62]

    Zorro: the masked multimodal transformer

    Adria Recasens, Jason Lin, Joao Carreira, Drew Jae- gle, Luyu Wang, Jean-baptiste Alayrac, Pauline Luc, An- toine Miech, Lucas Smaira, Ross Hemsley, et al. Zorro: the masked multimodal transformer. arXiv preprint arXiv:2301.09595, 2023. 1, 2

    2023 arXiv

  55. [63]

    Indoor segmentation and support inference from rgbd images

    Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV (5), 7576:746–760, 2012. 5

    2012

  56. [64]

    Mpnet: Masked and permuted pre-training for lan- guage understanding

    Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. Mpnet: Masked and permuted pre-training for lan- guage understanding. Advances in Neural Information Pro- cessing Systems, 33:16857–16867, 2020. 5

    2020

  57. [65]

    Sun rgb-d: A rgb-d scene understanding benchmark suite

    Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understanding benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 567–576, 2015. 5

    2015

  58. [66]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. 5

    2012 arXiv

  59. [67]

    Omnivec: Learn- ing robust representations with cross modal sharing

    Siddharth Srivastava and Gaurav Sharma. Omnivec: Learn- ing robust representations with cross modal sharing. arXiv preprint arXiv:2311.05709, 2023. 2, 4, 5, 6, 7

    2023 arXiv

  60. [68]

    Hierarchical multi-task learning via task affin- ity groupings

    Siddharth Srivastava, Swati Bhugra, Vinay Kaushik, and Brejesh Lall. Hierarchical multi-task learning via task affin- ity groupings. In 2023 IEEE International Conference on Image Processing (ICIP), pages 3289–3293. IEEE, 2023. 2

    2023

  61. [69]

    Benchmarking ro- bustness of 3d point cloud recognition against common cor- ruptions

    Jiachen Sun, Qingzhao Zhang, Bhavya Kailkhura, Zhiding Yu, Chaowei Xiao, and Z Morley Mao. Benchmarking ro- bustness of 3d point cloud recognition against common cor- ruptions. arXiv preprint arXiv:2201.12296, 2022. 6

    2022 arXiv

  62. [70]

    Efficientnet: Rethinking model scaling for convolutional neural networks

    Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In In- ternational conference on machine learning , pages 6105–

  63. [71]

    Contrastive boundary learning for point cloud segmentation

    Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 8489–8499, 2022. 6

    2022

  64. [72]

    Small sample hyper- spectral image classification based on the random patches network and recursive filtering

    Denis Uchaev and Dmitry Uchaev. Small sample hyper- spectral image classification based on the random patches network and recursive filtering. Sensors, 23(5):2499, 2023. 8

    2023

  65. [73]

    Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data

    Mikaela Angelina Uy, Quang-Hieu Pham, Binh-Son Hua, Thanh Nguyen, and Sai-Kit Yeung. Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data. In ICCV, pages 1588–1597,

  66. [74]

    The inaturalist species classification and detection dataset

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , page...

    2018

  67. [75]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 3

    2017

  68. [76]

    Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions

    Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vi- sion foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vi- si...

    2023

  69. [77]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191, 2022. 6

    2022 arXiv

  70. [78]

    Masked feature pre- diction for self-supervised visual pre-training

    Chen Wei, Haoqi Fan, Saining Xie, Chao-Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature pre- diction for self-supervised visual pre-training. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14668–14678, 2022. 2

    2022

  71. [79]

    Syn- cretic modality collaborative learning for visible infrared person re-identification

    Ziyu Wei, Xi Yang, Nannan Wang, and Xinbo Gao. Syn- cretic modality collaborative learning for visible infrared person re-identification. In ICCV, pages 225–234, 2021. 8

    2021

  72. [80]

    Controllable abstractive dialogue summarization with sketch supervision

    Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus Stene- torp, and Caiming Xiong. Controllable abstractive dialogue summarization with sketch supervision. arXiv preprint arXiv:2105.14064, 2021. 6

    2021 arXiv

  73. [81]

    Tinyvit: Fast pretraining distillation for small vision transformers

    Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan. Tinyvit: Fast pretraining distillation for small vision transformers. InEuropean Con- ference on Computer Vision, pages 68–85. Springer, 2022. 4

    2022

  74. [82]

    Point transformer v2: Grouped vector at- tention and partition-based pooling

    Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Heng- shuang Zhao. Point transformer v2: Grouped vector at- tention and partition-based pooling. Advances in Neural Information Processing Systems, 35:33330–33342, 2022. 6

    2022

  75. [83]

    3d shapenets: A deep representation for volumetric shapes

    Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015. 5

    1912

  76. [84]

    Audiovisual slow- fast networks for video recognition

    Fanyi Xiao, Yong Jae Lee, Kristen Grauman, Jitendra Malik, and Christoph Feichtenhofer. Audiovisual slow- fast networks for video recognition. arXiv preprint arXiv:2001.08740, 2020. 2

    2001 arXiv

  77. [85]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016. 5

    2016

  78. [86]

    Multimodal learning with transformers: A survey

    Peng Xu, Xiatian Zhu, and David A Clifton. Multimodal learning with transformers: A survey. arXiv preprint arXiv:2206.06488, 2022. 2

    2022 arXiv

  79. [87]

    Multi-modal masked pre-training for monocular panoramic depth completion

    Zhiqiang Yan, Xiang Li, Kun Wang, Zhenyu Zhang, Jun Li, and Jian Yang. Multi-modal masked pre-training for monocular panoramic depth completion. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I , pages 378–395. ...

    2022

  80. [88]

    Swin3d: A pretrained transformer backbone for 3d indoor scene understanding

    Yu-Qi Yang, Yu-Xiao Guo, Jian-Yu Xiong, Yang Liu, Hao Pan, Peng-Shuai Wang, Xin Tong, and Baining Guo. Swin3d: A pretrained transformer backbone for 3d indoor scene understanding. arXiv preprint arXiv:2304.06906 ,

    arXiv

  81. [89]

    generalized autoregressive pretraining for language understanding; 2019

    Z Yang, Z Dai, Y Yang, J Carbonell, RR Salakhutdinov, and XLNet Le QV . generalized autoregressive pretraining for language understanding; 2019. Preprint at https://arxiv. org/abs/1906.08237 Accessed June, 21, 2021. 5

    2019 arXiv

  82. [90]

    Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven C. H. Hoi. Deep learning for person re-identification: A survey and outlook. arXiv preprint arXiv:2001.04193, 2020. 8

    2001 arXiv

  83. [91]

    Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Process- ing Systems, 2021

    Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. Do transformers really perform badly for graph representation? In Thirty-Fifth Conference on Neural Information Process- ing Systems, 2021. 8

    2021

  84. [92]

    Coca: Con- trastive captioners are image-text foundation models

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Con- trastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022. 6

    2022 arXiv

  85. [93]

    Metaformer is actually what you need for vision

    Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10819–10829, 2022. 2, 5, 6, 8

    2022

  86. [94]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling

    Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 19313–19322, 2022. 2

    2022

  87. [95]

    So- cratic models: Composing zero-shot multimodal reasoning with language

    Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choro- manski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, et al. So- cratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022. 7

    2022 arXiv

  88. [96]

    Point- cutmix: Regularization strategy for point cloud classifica- tion

    Jinlai Zhang, Lyujie Chen, Bo Ouyang, Binbin Liu, Jihong Zhu, Yujin Chen, Yanmei Meng, and Danfeng Wu. Point- cutmix: Regularization strategy for point cloud classifica- tion. Neurocomputing, 505:58–67, 2022. 6

    2022

  89. [97]

    An overview of multi-task learning

    Yu Zhang and Qiang Yang. An overview of multi-task learning. National Science Review, 5(1):30–43, 2018. 2

    2018

  90. [98]

    Modality synergy complement learning with cascaded aggregation for visible-infrared person re- identification

    Yiyuan Zhang, Sanyuan Zhao, Yuhao Kang, and Jian- bing Shen. Modality synergy complement learning with cascaded aggregation for visible-infrared person re- identification. In Computer Vision–ECCV 2022: 17th Eu- ropean Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedi...

    2022

  91. [99]

    Meta- transformer: A unified framework for multimodal learning

    Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hong- sheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta- transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023. 2, 5, 7, 8

    2023 arXiv

  92. [100]

    Places: A 10 million image database for scene recognition

    Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analy- sis and machine intelligence, 40(6):1452–1464, 2017. 5

    2017

  93. [101]

    Scene parsing through ade20k dataset

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 633–641,

  94. [102]

    Locality-aware hyperspectral classification

    Fangqin Zhou, Mert Kilickaya, and Joaquin Vanschoren. Locality-aware hyperspectral classification. arXiv preprint arXiv:2309.01561, 2023. 8

    2023 arXiv

  95. [103]

    Informer: Be- yond efficient transformer for long sequence time-series forecasting

    Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. Informer: Be- yond efficient transformer for long sequence time-series forecasting. In AAAI, 2021. 5, 8

    2021

  96. [104]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI Conference on Arti- ficial Intelligence, 2018. 5

    2018

  97. [105]

    Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks

    Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre- training unified architecture for generic perception for zero- shot and few-shot tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.