Pith. sign in

REVIEW 3 major objections 4 minor 66 references

CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CM3AE is a self-supervised pretraining framework for RGB frames, event frames, and event voxels that claims the best scores on five event-vision downstream tasks.

desk verdict Sensible MAE extension to RGB-event data, but the unreported overlap between pre-training and evaluation sets undermines the headline results and must be resolved before this is trustworthy. read the letter →

arxiv 2504.12576 v1 pith:U2C7S5TQ submitted 2025-04-17 cs.CV cs.AI

Wentao Wu , Xiao Wang , Chenglong Li , Bo Jiang , Jin Tang , Bin Luo , Qi Liu This is my paper · ORCID
classification cs.CVcs.AI
keywords eventcameramaskedautoencodermultimodalpretrainingRGB-eventfusioncontrastivelearningactionrecognitionobjectdetectionvisualtracking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that a single self-supervised pretraining scheme can serve both event-only and RGB-event fusion models, and that it beats standard ImageNet-pretrained initialization on five event-vision benchmarks. To show this, it pretrains a dual-branch masked autoencoder on 2,535,759 paired RGB and event samples, adding a fusion-reconstruction head that rebuilds the RGB image from fused features and a contrastive loss that aligns RGB, event-frame, and event-voxel representations in a shared space. The reported results put CM3AE ahead of scratch training, ImageNet-pretrained baselines, and MAE pretrained on the same data on every metric, including larger gains when only 10-20% of downstream labels are available. If the comparison holds, event-vision researchers would no longer need ImageNet weights as a default starting point, and the fusion module itself could become a reusable component for RGB-event models.

What carries the argument

The load-bearing machinery is a dual-branch masked autoencoder in which 75% of patches are masked in both RGB and event frames while half of the unmasked patches are positionally shared between the two modalities; the shared positions prevent the fusion module from trivially recovering masked content from the other modality. On top of this, a multimodal fusion reconstruction module concatenates RGB encoder tokens with event tokens (and, in a second branch, with event-voxel tokens encoded by a small transformer) through a standard Transformer fusion block, then decodes the fused tokens to reconstruct the original RGB image. A multimodal contrastive loss treats RGB, event-frame, and event-voxel views of the same scene as positives and aligns them in a shared normalized space, complementing the local pixel-reconstruction objective with a global correspondence objective. The three losses are summed during pretraining, and the pretrained encoders, decoder, or fusion block are then transferred to downstream models.

What would settle it

Inspect the sequence identifiers of the 827,694 HARDVS pairs and 231,277 COESOT pairs in REV2M against the official HARDVS and COESOT test splits; any overlap means the reported action-recognition and tracking gains could be memorization. Retraining CM3AE and the MAE-on-REV2M baseline on the overlap-free corpus and comparing the gaps would settle the claim.

Watch

Extended reading notes

Core claim

The central claim is that pretraining on paired RGB and event data with a multimodal masked autoencoder yields transferable representations for both unimodal event tasks and RGB-event fusion tasks, and that the pretrained fusion block transfers separately. Concretely, on HARDVS action recognition the model reaches 53.40% Top-1 with RGB+Event input and 52.45% with event-only input, against 50.98% and 49.32% for MAE pretrained on the same REV2M corpus; on EvDET200K detection it reaches 49.0 AP against 47.5; on COESOT tracking it reaches 62.6 SR against 61.0; and on EventVOT it reaches 57.6 SR against 56.8. In few-shot settings with 10% of downstream training data, CM3AE reaches 43.68% Top-1 on RGB+Event action recognition, compared with 33.31% for MAE pretrained on REV2M and 5.63% from scratch. The paper also reports that loading only the pretrained fusion module into downstream RGB-event models improves action recognition Top-1 from 52.40% to 53.40% and tracking SR from 62.0 to 62.6.

Load-bearing premise

The load-bearing premise is that the REV2M pretraining corpus excludes the test splits of HARDVS and COESOT even though it draws 827,694 pairs from HARDVS and 231,277 pairs from COESOT, and those same datasets are then used as downstream benchmarks.

Editorial extensions

If this is right

  • Event-only models can be initialized from a backbone pretrained on paired RGB-event data; on every reported metric this beats both ImageNet-pretrained and REV2M-pretrained MAE initialization.
  • The pretrained fusion block transfers as a standalone component into RGB-event fusion models and improves them over the same architecture trained from scratch.
  • With only 10% or 20% of downstream labels, CM3AE initialization produces much larger gains than scratch or MAE-on-REV2M, indicating the pretraining reduces annotation requirements.
  • A single pretrained model serves both event-frame and event-voxel downstream inputs, so event-only and RGB-event models can share one foundation.
  • The REV2M corpus is assembled from five public datasets, so the pretraining recipe can be reproduced without new data collection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same recipe should transfer to other paired sensor modalities, such as RGB and thermal or RGB and depth, because the fusion-reconstruction and contrastive objectives are not event-specific; only the voxel encoder is.
  • Editorial inference: the position-sharing rule in the masking step is a candidate load-bearing detail: without it, the fusion module could exploit visible tokens from one modality to reconstruct masked positions of the other, so the masking alignment deserves a dedicated ablation.
  • Editorial inference: because only one pretraining corpus size is reported, the few-shot results invite a scaling study: if REV2M size is the active ingredient, larger corpora should push the same five benchmarks further, and smaller ones should shrink the gap to MAE.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes CM3AE, a self-supervised pre-training framework for RGB frames, event frames, and event voxels. It builds on a dual-branch masked autoencoder, adds a multimodal fusion reconstruction module that reconstructs RGB images from fused RGB/event/voxel features, and adds multimodal contrastive losses between RGB-event and RGB-voxel pairs. The authors construct REV2M, a corpus of 2,535,759 RGB-event pairs assembled from HARDVS, N-ImageNet, COESOT, VisEvent, and DSEC-MOD, pre-train a ViT-B/16 backbone, and evaluate on five downstream tasks: HARDVS action recognition in both RGB+Event and Event settings, EvDET200K object detection, COESOT RGB-Event tracking, and EventVOT event tracking. The central empirical claim is that CM3AE outperforms scratch, ImageNet-pretrained baselines, and MAE pre-trained on the same REV2M data on every reported metric.

Significance. If the evaluation is leakage-free, the paper would provide a useful RGB-event pre-training recipe with a clearly described architecture and informative ablations, and the promised release of code and pretrained models would facilitate reproducibility. The main concern is that the evaluation protocol is not currently established: the pre-training corpus includes datasets used for downstream evaluation and the paper does not document exclusion of test splits. Because the self-supervised objectives do not use downstream labels, there is no derivation-level circularity, but the transfer claim depends on split hygiene. The stress-test concern about HARDVS and COESOT overlap is directly load-bearing and must be resolved before the empirical claims can be accepted.

major comments (3)
  1. [§4.1, §4.3, Table 1] REV2M is constructed from HARDVS (827,694 pairs) and COESOT (231,277 pairs), and the same two datasets are used as downstream benchmarks for action recognition and RGB-Event tracking. The paper never states that the official test splits of HARDVS and COESOT were excluded from pre-training; for COESOT, the text itself reports an official split of 827 training and 527 test videos, so using the full released corpus would place test sequences in pre-training. Because a masked autoencoder can memorize scene structure, the reported gains over scratch, ImageNet-pretrained baselines, and MAE-on-REV2M would be inflated or invalid under such overlap. Please document the exact split construction and any deduplication, and re-run the comparisons with pre-training strictly on training splits or on data disjoint from all downstream test sequences.
  2. [§4.3, Tables 1 and 2] There is a direct numerical inconsistency in the headline results. Table 1 reports Our REV2M as 53.40/65.82 for RGB+Event action recognition and 52.45/63.76 for Event action recognition, while the text states that RGB-Event accuracy improved to 53.24% and 65.13% and Event accuracy to 53.18% and 62.88%; these latter values match the full ablation row in Table 2. Please reconcile the tables and text and report which configuration produced Table 1, since these numbers are the central evidence for the paper's main claim.
  3. [§4.3, Tables 1 and 4] No error bars or multiple-seed results are reported, and several comparisons in Table 1 are within about one point (for example, EventVOT SR 57.6 vs 56.8 for MAE on REV2M, and COESOT NPR 72.8 vs 71.5). Given fine-tuning stochasticity, the claim that CM3AE achieves the best performance on every metric needs variance estimates or at least repeated runs with different seeds to establish that the observed gaps are not noise.
minor comments (4)
  1. [§4.3] The statement that CM3AE achieves superior performance with only 25% of MAE's training time is not supported by any training schedule or timing table in the manuscript; please provide the basis for this comparison.
  2. [§3.2] The text says the authors 'first reconstruct the original images of each modality in an autoregressive manner,' but the masked autoencoder described in the same section is not autoregressive; please rephrase to avoid a terminology conflict.
  3. [§4.1] The dataset name 'Visevent' in Section 4.1 is inconsistent with 'VisEvent' in the reference list and elsewhere; please standardize the spelling, and also fix the recurring 'V oxel' spacing and 'multimodel' typo.
  4. [Figure 2 and Table 5] Figure 2's caption refers to a 'multi-modal fusion generation module' while the text calls it a 'multimodal fusion reconstruction module,' and Table 5's 'w/o Pre-training' condition is unclear about whether the fusion module is randomly initialized while the rest of the backbone uses pretrained weights or whether the whole model is trained from scratch; please clarify both.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the pre-training losses are self-supervised, the downstream baselines are external, and the REV2M/HARDVS-COESOT overlap is a data-leakage concern rather than a derivation-circle.

full rationale

CM3AE's derivation chain is self-contained and externally benchmarked. The pre-training objectives (Eq. 1, Eq. 2, and Eqs. 5-8) are masked-reconstruction and cross-modal contrastive losses defined over paired RGB, event-frame, and event-voxel inputs; they contain no downstream task labels and no fitted parameter that is later renamed as a prediction. The downstream comparisons in Table 1 are made against scratch, ImageNet-pretrained models, and MAE pretrained on the same REV2M data, so the claimed improvement is an empirical result rather than a consequence of the loss definitions. Citations to the authors' earlier datasets (HARDVS [56], COESOT [44], VisEvent [51], EvDET200K [52], EventVOT [54]) are used as data sources and benchmarks, not as uniqueness theorems or ansatz justifications, and none of those citations carries the derivation. I do flag one evaluation-validity concern, but it is not circular: Section 4.1 states that REV2M is 'sourced from five publicly available datasets: HARDVS [56], N-ImageNet [21], COESOT [44], Visevent [51] and DSEC-MOD [64]' and then lists HARDVS and COESOT as downstream benchmarks, without stating that their official test splits were excluded from REV2M. If test frames were included, the reported gains on those benchmarks would be inflated by memorization rather than transfer; that is a data-leakage risk, not a derivation that reduces to its own input. Under the circularity rubric, the paper receives 0.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The method introduces no new physical entities. It relies on standard transformer and contrastive learning machinery plus a few hand-chosen hyperparameters (mask ratio, voxel count, contrastive scale, loss weights). The most significant unstated assumption is that RGB and event frames are spatially aligned, and the most consequential practical risk is the overlap between pre-training and evaluation datasets.

free parameters (4)
  • mask_ratio = 0.75
    Selected via ablation on HARDVS (Table 3). The final model uses 75% masking; the paper fits this hyperparameter on the downstream task domain.
  • fixed_voxel_count_per_image = not specified
    Section 3.2 states each image uses a fixed number of voxels with sampling to fill or truncate, but the number is never given. This is a hand-chosen design parameter that affects the voxel encoder input.
  • contrastive_scale_ls = not specified
    Introduced in Eq. (4) as the scaling factor for contrastive logits; no value or schedule is reported.
  • loss_weights = all 1.0 in Eq. (8)
    The total loss adds Lm, Lf, and Lcl with equal weight; no tuning is reported, so relative weighting is an implicit design choice.
assumptions (4)
  • domain assumption Paired RGB and event frames are spatially aligned so that patch indices correspond across modalities.
    The fusion module and the shared-position masking rule in Section 3.2 assume corresponding patches observe the same scene location. This holds only if the camera setup is calibrated; no alignment step is described.
  • domain assumption Event voxel representation with 14 points per voxel and fixed-size sampling preserves sufficient temporal information.
    Section 3.2 introduces this representation without evaluating alternatives or specifying the fixed voxel count; downstream gains are attributed to it in the fusion module.
  • ad hoc to paper Reconstructing the RGB image from fused multimodal features forces the model to learn complementary cross-modal information.
    This is the paper's central design hypothesis behind MFRM; it is plausible but is a modeling assumption not derived from prior theory.
  • standard math Transformer encoder-decoder and contrastive learning frameworks are valid for self-supervised representation learning.
    The method rests on standard ML architecture and loss functions from prior work (MAE, CLIP); accepted as background.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework." pith.science (2026). https://pith.science/paper/U2C7S5TQ

@misc{pith2026250412576,
  author       = {Pith},
  title        = {Pith review of: CM3AE: A Unified RGB Frame and Event-Voxel/-Frame Pre-training Framework},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2C7S5TQ}},
  note         = {Machine review of arXiv:2504.12576}
}
read the original abstract

Event cameras have attracted increasing attention in recent years due to their advantages in high dynamic range, high temporal resolution, low power consumption, and low latency. Some researchers have begun exploring pre-training directly on event data. Nevertheless, these efforts often fail to establish strong connections with RGB frames, limiting their applicability in multi-modal fusion scenarios. To address these issues, we propose a novel CM3AE pre-training framework for the RGB-Event perception. This framework accepts multi-modalities/views of data as input, including RGB images, event images, and event voxels, providing robust support for both event-based and RGB-event fusion based downstream tasks. Specifically, we design a multi-modal fusion reconstruction module that reconstructs the original image from fused multi-modal features, explicitly enhancing the model's ability to aggregate cross-modal complementary information. Additionally, we employ a multi-modal contrastive learning strategy to align cross-modal feature representations in a shared latent space, which effectively enhances the model's capability for multi-modal understanding and capturing global dependencies. We construct a large-scale dataset containing 2,535,759 RGB-Event data pairs for the pre-training. Extensive experiments on five downstream tasks fully demonstrated the effectiveness of CM3AE. Source code and pre-trained models will be released on https://github.com/Event-AHU/CM3AE.

Figures

Figures reproduced from arXiv: 2504.12576 by the authors.

Figure 1
Figure 1. Our proposed pre-trained CM3AE model takes RGB-Event Streams/-Frames as the input and supports unimodal and multimodal [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An overview of our proposed pre-training framework for RGB-Event stream perceptron, termed CM3AE. Specifically, the [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Representative samples in our pre-training dataset. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Visualization of the experimental results of each downstream task. [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Visualization of attention maps on different downstream tasks. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Visualization of reconstructed RGB and Event images [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

66 extracted references · 34 canonical work pages

  1. [1]

    Multimae: Multi-modal multi-task masked autoen- coders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoen- coders. In European Conference on Computer Vision, pages 348–367. Springer, 2022. 1, 3

    work page 2022

  2. [2]

    Beit: Bert pre-training of image transformers

    Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference on Learning Representations, 2022. 3, 7, 8

    work page 2022

  3. [3]

    Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? arXiv preprint arXiv:2102.05095, 2021. 6, 8

    arXiv 2021

  4. [4]

    Deepseek llm: Scaling open- source language models with longtermism

    Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. Deepseek llm: Scaling open- source language models with longtermism. arXiv preprint arXiv:2401.02954, 2024. 1

    arXiv 2024

  5. [5]

    Lan- guage models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners. Advances in neural in- formation processing systems, 33:1877–1901, 2020. 1

    1901

  6. [6]

    End-to- end object detection with transformers

    Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to- end object detection with transformers. In European confer- ence on computer vision, pages 213–229. Springer, 2020. 3

    2020

  7. [7]

    Weakly misalignment-free adaptive feature alignment for uavs- based multimodal object detection

    Chen Chen, Jiahao Qi, Xingyue Liu, Kangcheng Bin, Ruigang Fu, Xikun Hu, and Ping Zhong. Weakly misalignment-free adaptive feature alignment for uavs- based multimodal object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26836–26845, 2024. 3

    work page 2024

  8. [8]

    An empirical study of training self-supervised vision transformers

    Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. InPro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9640–9649, 2021. 7, 8

    work page 2021

Show all 66 references
  1. [9]

    Segment any event streams via 11 weighted adaptation of pivotal tokens

    Zhiwen Chen, Zhiyu Zhu, Yifan Zhang, Junhui Hou, Guang- ming Shi, and Jinjian Wu. Segment any event streams via 11 weighted adaptation of pivotal tokens. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3890–3900, 2024. 1

    2024

  2. [10]

    Unihcp: A unified model for human-centric perceptions

    Yuanzheng Ci, Yizhou Wang, Meilin Chen, Shixiang Tang, Lei Bai, Feng Zhu, Rui Zhao, Fengwei Yu, Donglian Qi, and Wanli Ouyang. Unihcp: A unified model for human-centric perceptions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 178...

    2023

  3. [11]

    Satmae: Pre-training transformers for tem- poral and multi-spectral satellite imagery

    Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for tem- poral and multi-spectral satellite imagery. Advances in Neu- ral Information Processing Systems, 35:197–211, 2022. 3

    2022

  4. [12]

    Deepseekmoe: Towards ultimate expert spe- cialization in mixture-of-experts language models

    Damai Dai, Chengqi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li, Wangding Zeng, Xingkai Yu, Y Wu, et al. Deepseekmoe: Towards ultimate expert spe- cialization in mixture-of-experts language models. CoRR,

  5. [13]

    Sfod: Spiking fusion object detector

    Yimeng Fan, Wei Zhang, Changsong Liu, Mingyang Li, and Wenrui Lu. Sfod: Spiking fusion object detector. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17191–17200, 2024. 3

    2024

  6. [14]

    Hypergraph-based multi-view action recognition using event cameras

    Yue Gao, Jiaxuan Lu, Siqi Li, Yipeng Li, and Shaoyi Du. Hypergraph-based multi-view action recognition using event cameras. IEEE Transactions on Pattern Analysis and Ma- chine Intelligence, 2024. 4

    2024

  7. [15]

    Multimodal masked autoen- coders learn transferable representations

    Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, and Pieter Abbeel. Multimodal masked autoen- coders learn transferable representations. arXiv preprint arXiv:2205.14204, 2022. 3

    2022 arXiv

  8. [16]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Ab- hinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024. 1

    2024 arXiv

  9. [17]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025. 1

    2025 arXiv

  10. [18]

    Momentum contrast for unsupervised visual rep- resentation learning

    Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual rep- resentation learning. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , pages 9729–9738, 2020. 3

    2020

  11. [19]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000– 16009, 2022. 1, 3, 7, 8

    2022

  12. [20]

    Data-efficient event cam- era pre-training via disentangled masked modeling

    Zhenpeng Huang, Chao Li, Hao Chen, Yongjian Deng, Yifeng Geng, and Limin Wang. Data-efficient event cam- era pre-training via disentangled masked modeling. arXiv preprint arXiv:2403.00416, 2024. 2

    2024 arXiv

  13. [21]

    N-imagenet: Towards robust, fine-grained object recognition with event cameras

    Junho Kim, Jaehyeok Bae, Gangin Park, Dongsu Zhang, and Young Min Kim. N-imagenet: Towards robust, fine-grained object recognition with event cameras. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2146–2156, 2021. 7

    2021

  14. [22]

    Spiking-yolo: spiking neural network for energy- efficient object detection

    Seijoon Kim, Seongsik Park, Byunggook Na, and Sungroh Yoon. Spiking-yolo: spiking neural network for energy- efficient object detection. In Proceedings of the AAAI con- ference on artificial intelligence, pages 11270–11277, 2020. 3

    2020

  15. [23]

    Segment any- thing

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF international confer- ence on computer vision, pages 4015–4026, 2023. 1

    2023

  16. [24]

    Masked event modeling: Self-supervised pretraining for event cameras

    Simon Klenk, David Bonello, Lukas Koestler, Nikita Araslanov, and Daniel Cremers. Masked event modeling: Self-supervised pretraining for event cameras. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2378–2388, 2024. 2

    2024

  17. [25]

    Openess: Event-based semantic scene understanding with open vocabularies

    Lingdong Kong, Youquan Liu, Lai Xing Ng, Benoit R Cot- tereau, and Wei Tsang Ooi. Openess: Event-based semantic scene understanding with open vocabularies. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 15686–15698, 2024. 1

    2024

  18. [26]

    Mulfs-cap: Multimodal fusion- supervised cross-modality alignment perception for unreg- istered infrared-visible image fusion

    Huafeng Li, Zengyi Yang, Yafei Zhang, Wei Jia, Zheng- tao Yu, and Yu Liu. Mulfs-cap: Multimodal fusion- supervised cross-modality alignment perception for unreg- istered infrared-visible image fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025. 3

    2025

  19. [27]

    Coupled mamba: Enhanced multimodal fusion with coupled state space model

    Wenbing Li, Hang Zhou, Junqing Yu, Zikai Song, and Wei Yang. Coupled mamba: Enhanced multimodal fusion with coupled state space model. InNeural Information Processing Systems, 2024. 3

    2024

  20. [28]

    Exploring plain vision transformer backbones for object de- tection

    Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object de- tection. In European conference on computer vision , pages 280–296. Springer, 2022. 3, 6

    2022

  21. [29]

    Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model

    Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, 2024. 1

    2024 arXiv

  22. [30]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892–34916, 2023. 1

    2023

  23. [31]

    Pixmim: Rethinking pixel reconstruction in masked image modeling

    Yuan Liu, Songyang Zhang, Jiacheng Chen, Kai Chen, and Dahua Lin. Pixmim: Rethinking pixel reconstruction in masked image modeling. Transactions on Machine Learn- ing Research. 7, 8

  24. [32]

    Decoupled weight de- cay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight de- cay regularization. In International Conference on Learning Representations, 2019. 8

    2019

  25. [33]

    Integer-valued training and spike-driven inference spiking neural network for high-performance and energy-efficient object detection

    Xinhao Luo, Man Yao, Yuhong Chou, Bo Xu, and Guoqi Li. Integer-valued training and spike-driven inference spiking neural network for high-performance and energy-efficient object detection. In European Conference on Computer Vi- sion, pages 253–272. Springer, 2024. 3

    2024

  26. [34]

    Event-based moving object 12 detection and tracking

    Anton Mitrokhin, Cornelia Ferm ¨uller, Chethan Paramesh- wara, and Yiannis Aloimonos. Event-based moving object 12 detection and tracking. In2018 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), pages 1–9. IEEE, 2018. 1

    2018

  27. [35]

    Rethinking transformers pre-training for multi- spectral satellite imagery

    Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Fahad Shah- baz Khan. Rethinking transformers pre-training for multi- spectral satellite imagery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , page...

    2024

  28. [36]

    Dinov2: Learning robust visual features without supervision

    Maxime Oquab, Timoth ´ee Darcet, Th ´eo Moutakanni, Huy V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Jou...

    2024

  29. [37]

    Pytorch: An im- perative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An im- perative style, high-performance deep learning library. Ad- vances in neural information processing systems ...

    2019

  30. [38]

    Beit v2: Masked image modeling with vector-quantized visual tokenizers

    Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208.06366, 2022. 7, 8

    2022 arXiv

  31. [39]

    Detectors: Detecting objects with recursive feature pyramid and switch- able atrous convolution

    Siyuan Qiao, Liang-Chieh Chen, and Alan Yuille. Detectors: Detecting objects with recursive feature pyramid and switch- able atrous convolution. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10213–10224, 2021. 3

    2021

  32. [40]

    Improving language understanding by gen- erative pre-training

    Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by gen- erative pre-training. 2018. 1

    2018

  33. [41]

    Language models are unsu- pervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsu- pervised multitask learners. OpenAI blog, 1(8):9, 2019. 1

    2019

  34. [42]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  35. [43]

    Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning

    Colorado J Reed, Ritwik Gupta, Shufan Li, Sarah Brock- man, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. In Proceedings of th...

    2023

  36. [44]

    Revisiting color-event based tracking: A unified network, dataset, and metric

    Chuanming Tang, Xiao Wang, Ju Huang, Bo Jiang, Lin Zhu, Jianlin Zhang, Yaowei Wang, and Yonghong Tian. Revisiting color-event based tracking: A unified network, dataset, and metric. arXiv preprint arXiv:2211.11010, 2022. 6, 7

    2022 arXiv

  37. [45]

    Humanbench: Towards general human- centric perception with projector assisted pretraining

    Shixiang Tang, Cheng Chen, Qingsong Xie, Meilin Chen, Yizhou Wang, Yuanzheng Ci, Lei Bai, Feng Zhu, Haiyang Yang, Li Yi, et al. Humanbench: Towards general human- centric perception with projector assisted pretraining. InPro- ceedings of the IEEE/CVF Conference on Computer Vis...

    2023

  38. [46]

    Videomae: Masked autoencoders are data-efficient learn- ers for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learn- ers for self-supervised video pre-training. arXiv preprint arXiv:2203.12602, 2022. 3

    2022 arXiv

  39. [47]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 1

    2023 arXiv

  40. [48]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 1

    2023 arXiv

  41. [49]

    Videomae v2: Scaling video masked autoencoders with dual masking

    Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yi- nan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 14549–14560, 2023. 3

    2023

  42. [50]

    Image as a foreign language: Beit pretraining for vision and vision- language tasks

    Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhil- iang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mo- hammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision- language tasks. In Proceedings of the IEEE/CVF Conferen...

    2023

  43. [51]

    Vi- sevent: Reliable object tracking via collaboration of frame and event flows

    Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu. Vi- sevent: Reliable object tracking via collaboration of frame and event flows. IEEE Transactions on Cybernetics, 54(3): 1997–2010, 2023. 7

    1997

  44. [52]

    Object detection using event camera: A moe heat conduction based detector and a new benchmark dataset

    Xiao Wang, Yu Jin, Wentao Wu, Wei Zhang, Lin Zhu, Bo Jiang, and Yonghong Tian. Object detection using event camera: A moe heat conduction based detector and a new benchmark dataset. arXiv preprint arXiv:2412.06647, 2024. 3, 7

    2024 arXiv

  45. [53]

    Pre-training on high definition x-ray images: An experimental study

    Xiao Wang, Yuehang Li, Wentao Wu, Jiandong Jin, Yao Rong, Bo Jiang, Chuanfu Li, and Jin Tang. Pre-training on high definition x-ray images: An experimental study. arXiv preprint arXiv:2404.17926, 2024. 3

    2024 arXiv

  46. [54]

    Event stream-based visual object tracking: A high-resolution benchmark dataset and a novel baseline

    Xiao Wang, Shiao Wang, Chuanming Tang, Lin Zhu, Bo Jiang, Yonghong Tian, and Jin Tang. Event stream-based visual object tracking: A high-resolution benchmark dataset and a novel baseline. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, p...

    2024

  47. [55]

    Structural information guided multimodal pre-training for vehicle-centric percep- tion

    Xiao Wang, Wentao Wu, Chenglong Li, Zhicheng Zhao, Zhe Chen, Yukai Shi, and Jin Tang. Structural information guided multimodal pre-training for vehicle-centric percep- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5624–5632, 2024. 3

    2024

  48. [56]

    Hardvs: Re- visiting human activity recognition with dynamic vision sen- sors

    Xiao Wang, Zongzhen Wu, Bo Jiang, Zhimin Bao, Lin Zhu, Guoqi Li, Yaowei Wang, and Yonghong Tian. Hardvs: Re- visiting human activity recognition with dynamic vision sen- sors. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 5615–5623, 2024. 1, 4, 7 13

    2024

  49. [57]

    Multipath event-based network for low-power human action recognition

    Xiao Wu and Junsong Yuan. Multipath event-based network for low-power human action recognition. In 2020 IEEE 6th World Forum on Internet of Things (WF-IoT) , pages 1–5. IEEE, 2020. 3

    2020

  50. [58]

    Event camera data pre-training

    Yan Yang, Liyuan Pan, and Liu Liu. Event camera data pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10699–10709, 2023. 2

    2023

  51. [59]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021. 3

    2021 arXiv

  52. [60]

    Dino: Detr with improved denoising anchor boxes for end-to-end object de- tection

    Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung-Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object de- tection. In The Eleventh International Conference on Learn- ing Representations, 2022. 1, 7, 8

    2022

  53. [61]

    Odtrack: Online dense temporal token learning for visual tracking

    Yaozong Zheng, Bineng Zhong, Qihua Liang, Zhiyi Mo, Shengping Zhang, and Xianxian Li. Odtrack: Online dense temporal token learning for visual tracking. In Proceed- ings of the AAAI conference on artificial intelligence , pages 7588–7596, 2024. 6

    2024

  54. [62]

    Image bert pre-training with online tokenizer

    Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image bert pre-training with online tokenizer. In International Conference on Learning Representations, 2021. 1, 7, 8

    2021

  55. [63]

    Ex- act: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more

    Jiazhou Zhou, Xu Zheng, Yuanhuiyi Lyu, and Lin Wang. Ex- act: Language-guided conceptual reasoning and uncertainty estimation for event-based action recognition and more. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, pages 18633–18643, 2024. 4

    2024

  56. [64]

    Event-free moving object segmentation from moving ego vehicle

    Zhuyun Zhou, Zongwei Wu, Danda Pani Paudel, R´emi Bout- teau, Fan Yang, Luc Van Gool, Radu Timofte, and Do- minique Ginhac. Event-free moving object segmentation from moving ego vehicle. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages ...

    2024

  57. [65]

    Segment everything everywhere all at once

    Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Wang, Lijuan Wang, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. Advances in neural information processing systems, 36:19769–19782,

  58. [66]

    Plip: Language-image pre-training for person representation learning

    Jialong Zuo, Changqian Yu, Nong Sang, and Changxin Gao. Plip: Language-image pre-training for person representation learning. arXiv preprint arXiv:2305.08386, 2023. 3 14

    2023 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.