{"id":"08123f1f-4e9b-464d-9793-7f6cee642013","arxiv_id":"2504.12576","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CM3AE combines masked autoencoding, cross-modal fusion reconstruction, and contrastive learning to pre-train a ViT on 2.53 million RGB-event pairs.","lead":"This paper introduces CM3AE, a self-supervised pre-training method that learns joint representations from paired RGB images, event frames, and event voxels. It reports consistent gains on five event-camera tasks, although the evaluation is weakened by pre-training on datasets that also serve as downstream test beds.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper does not document excluding HARDVS and COESOT test splits from the REV2M pre-training corpus, so downstream gains on those benchmarks may reflect memorization instead of transfer.","rationale":"The reader identified the same load-bearing weakness: REV2M is constructed from HARDVS and COESOT, both of which are used as downstream evaluation benchmarks, and the paper provides no explicit statement that test sequences were excluded. This is the correct primary concern because the paper's central empirical claim is about transfer: pre-training on a large RGB-event corpus should improve performance on downstream tasks. If the pre-training corpus contains the evaluation sequences, the reported gains are not evidence of transfer. The concern is not merely about protocol convention; it is about whether the measured effect is memorization rather than representation learning. I considered other potential weaknesses, such as the absence of directly comparable event pre-training baselines and the lack of released code or error bars, but those affect the strength or reproducibility of the contribution rather than the validity of the central comparison. The train/test overlap issue is more fundamental: if it lands, the main experimental claim no longer supports the paper's conclusion. The proposed test is straightforward and decisive: inspecting the REV2M construction script or dataset manifest and intersecting sequence IDs with official test splits, then rerunning the affected comparisons on a disjoint pre-training corpus. Because the current verdict 'CONDITIONAL' already makes acceptance contingent on resolving exactly this issue, my read does not move the verdict; it reinforces the condition.","tokens_in":1016,"tokens_out":967,"duration_ms":52867,"concrete_test":"Obtain the REV2M construction manifest or script and compute a sequence-level intersection between the pre-training sources and the downstream test splits. For COESOT, compare the video IDs in REV2M's 231,277 pairs against the official 527 test sequences; for HARDVS, compare against the official held-out test clips or class-disjoint split. If any test sequence appears in REV2M, remove those sequences from pre-training and rerun the key Table 1 comparisons; if the CM3AE gains over MAE-on-REV2M and ImageNet baselines shrink materially or collapse, the reported effectiveness is substantially due to pre-training on evaluation data. If no overlap exists, rerun one headline comparison with a fixed seed to confirm the reported margins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim in Section 4.3 is that CM3AE, after self-supervised pre-training, achieves the best results on five downstream tasks compared with scratch, ImageNet-pretrained baselines, and MAE pre-trained on the same REV2M data. For this claim to support a transferable pre-training method, the evaluation must be on sequences unseen during pre-training. Section 4.1 states that REV2M is built from HARDVS (827,694 pairs), N-ImageNet, COESOT (231,277 pairs), VisEvent, and DSEC-MOD. The same section then lists HARDVS and COESOT as downstream benchmarks, and Section 4.3 reports action recognition on HARDVS and visual object tracking on COESOT. The paper never states that test sequences were excluded from REV2M. This matters concretely: for COESOT, the paper itself gives an official split of 827 training and 527 test videos; if all 231,277 COESOT pairs were used for pre-training, the test videos are almost certainly included. For HARDVS, if the full released corpus was used without the official train/test split, pre-training also sees the evaluation clips. A masked autoencoder can memorize appearance and scene structure, so a model exposed to test frames during pre-training can inflate action recognition and tracking results without learning transferable cross-modal representations. The comparison against MAE-on-REV2M is also affected, since both models would see the same leaked data; the headline 'best on every metric' would then be a ranking under leakage, not evidence of a general foundation model. The omission is a concrete, checkable gap rather than an internal inconsistency, but it is load-bearing because it determines whether the empirical core of the paper supports the claimed generalization.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CM3AE, a self-supervised pre-training framework for RGB frames, event frames, and event voxels. It builds on a dual-branch masked autoencoder, adds a multimodal fusion reconstruction module that reconstructs RGB images from fused RGB/event/voxel features, and adds multimodal contrastive losses between RGB-event and RGB-voxel pairs. The authors construct REV2M, a corpus of 2,535,759 RGB-event pairs assembled from HARDVS, N-ImageNet, COESOT, VisEvent, and DSEC-MOD, pre-train a ViT-B/16 backbone, and evaluate on five downstream tasks: HARDVS action recognition in both RGB+Event and Event settings, EvDET200K object detection, COESOT RGB-Event tracking, and EventVOT event tracking. The central empirical claim is that CM3AE outperforms scratch, ImageNet-pretrained baselines, and MAE pre-trained on the same REV2M data on every reported metric.","tokens_in":17273,"tokens_out":5687,"duration_ms":58748,"significance":"If the evaluation is leakage-free, the paper would provide a useful RGB-event pre-training recipe with a clearly described architecture and informative ablations, and the promised release of code and pretrained models would facilitate reproducibility. The main concern is that the evaluation protocol is not currently established: the pre-training corpus includes datasets used for downstream evaluation and the paper does not document exclusion of test splits. Because the self-supervised objectives do not use downstream labels, there is no derivation-level circularity, but the transfer claim depends on split hygiene. The stress-test concern about HARDVS and COESOT overlap is directly load-bearing and must be resolved before the empirical claims can be accepted.","major_comments":[{"comment":"REV2M is constructed from HARDVS (827,694 pairs) and COESOT (231,277 pairs), and the same two datasets are used as downstream benchmarks for action recognition and RGB-Event tracking. The paper never states that the official test splits of HARDVS and COESOT were excluded from pre-training; for COESOT, the text itself reports an official split of 827 training and 527 test videos, so using the full released corpus would place test sequences in pre-training. Because a masked autoencoder can memorize scene structure, the reported gains over scratch, ImageNet-pretrained baselines, and MAE-on-REV2M would be inflated or invalid under such overlap. Please document the exact split construction and any deduplication, and re-run the comparisons with pre-training strictly on training splits or on data disjoint from all downstream test sequences.","section":"§4.1, §4.3, Table 1"},{"comment":"There is a direct numerical inconsistency in the headline results. Table 1 reports Our REV2M as 53.40/65.82 for RGB+Event action recognition and 52.45/63.76 for Event action recognition, while the text states that RGB-Event accuracy improved to 53.24% and 65.13% and Event accuracy to 53.18% and 62.88%; these latter values match the full ablation row in Table 2. Please reconcile the tables and text and report which configuration produced Table 1, since these numbers are the central evidence for the paper's main claim.","section":"§4.3, Tables 1 and 2"},{"comment":"No error bars or multiple-seed results are reported, and several comparisons in Table 1 are within about one point (for example, EventVOT SR 57.6 vs 56.8 for MAE on REV2M, and COESOT NPR 72.8 vs 71.5). Given fine-tuning stochasticity, the claim that CM3AE achieves the best performance on every metric needs variance estimates or at least repeated runs with different seeds to establish that the observed gaps are not noise.","section":"§4.3, Tables 1 and 4"}],"minor_comments":[{"comment":"The statement that CM3AE achieves superior performance with only 25% of MAE's training time is not supported by any training schedule or timing table in the manuscript; please provide the basis for this comparison.","section":"§4.3"},{"comment":"The text says the authors 'first reconstruct the original images of each modality in an autoregressive manner,' but the masked autoencoder described in the same section is not autoregressive; please rephrase to avoid a terminology conflict.","section":"§3.2"},{"comment":"The dataset name 'Visevent' in Section 4.1 is inconsistent with 'VisEvent' in the reference list and elsewhere; please standardize the spelling, and also fix the recurring 'V oxel' spacing and 'multimodel' typo.","section":"§4.1"},{"comment":"Figure 2's caption refers to a 'multi-modal fusion generation module' while the text calls it a 'multimodal fusion reconstruction module,' and Table 5's 'w/o Pre-training' condition is unclear about whether the fusion module is randomly initialized while the rest of the backbone uses pretrained weights or whether the whole model is trained from scratch; please clarify both.","section":"Figure 2 and Table 5"}],"recommendation":"major_revision","confidential_remarks":"The overlap between REV2M and the downstream benchmarks is the most serious issue in this submission. If test sequences from HARDVS or COESOT were included in pre-training, the headline comparisons would not support the abstract's claims. I am recommending major revision rather than rejection because the proposed framework itself appears reasonable and the issue could be fixed by documenting and enforcing proper split separation, and then re-running the affected experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper extends MAE-style pre-training to paired RGB and event data and builds the largest RGB-event pre-training corpus I know of (2.53M pairs). But the evaluation has a load-bearing leak: REV2M is built from HARDVS and COESOT, the same datasets used for downstream evaluation, and the paper never states that test sequences were excluded. On the face of it, the COESOT test split (527 of 1,354 videos) is almost certainly inside the 231,277 pairs taken from that dataset. If those sequences appear during pre-training, the headline 'best on every metric' is largely memorization, not transfer.\n\nThe architecture is described clearly: dual-branch MAE with 75% masking, a fusion reconstruction head, and contrastive alignment between RGB-event and RGB-voxel pairs. The ablations in Table 2 support each added component. That part is fine.\n\nThe soft spots are real and concrete. First, the split issue. Section 4.1 lists HARDVS and COESOT as pre-training sources and then uses exactly those datasets as downstream benchmarks. The authors do not say they held out the official test videos. Second, the ablations themselves are circular: Section 4.4 says they pre-train on HARDVS and then evaluate action recognition on HARDVS. That makes the internal comparisons in Tables 2 and 3 vacuous as evidence of generalization, and it casts doubt on the main numbers. Third, the paper cites but does not compare against the closest event pre-training works (EventMAE, disentangled masked modeling), so we don't know whether the new combination buys anything over existing event-specific recipes. Fourth, there are no error bars, and the few-shot results in Table 4 are striking enough that a single run would worry me.\n\nI'd send this to a serious referee, but the referee should demand a clear statement on test-split exclusion, re-run the ablations with a proper held-out set, and add the missing baselines. If the leak is confirmed, the central claim collapses to 'pre-training on the test set helps,' which is not a result worth keeping.\n\nFor your own reading, the dataset alone, if properly split, could be useful; the current version is not citable.","headline":"Sensible MAE extension to RGB-event data, but the unreported overlap between pre-training and evaluation sets undermines the headline results and must be resolved before this is trustworthy.","tokens_in":17856,"tokens_out":3717,"would_cite":false,"duration_ms":33391,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CM3AE is a self-supervised pretraining framework for RGB frames, event frames, and event voxels that claims the best scores on five event-vision downstream tasks.","keywords":["event camera","masked autoencoder","multimodal pretraining","RGB-event fusion","contrastive learning","action recognition","object detection","visual object tracking"],"falsifier":"Inspect the sequence identifiers of the 827,694 HARDVS pairs and 231,277 COESOT pairs in REV2M against the official HARDVS and COESOT test splits; any overlap means the reported action-recognition and tracking gains could be memorization. Retraining CM3AE and the MAE-on-REV2M baseline on the overlap-free corpus and comparing the gaps would settle the claim.","tokens_in":16739,"feed_emoji":"⚡","tokens_out":11294,"duration_ms":104834,"temperature":0.7,"pith_summary":"The paper argues that a single self-supervised pretraining scheme can serve both event-only and RGB-event fusion models, and that it beats standard ImageNet-pretrained initialization on five event-vision benchmarks. To show this, it pretrains a dual-branch masked autoencoder on 2,535,759 paired RGB and event samples, adding a fusion-reconstruction head that rebuilds the RGB image from fused features and a contrastive loss that aligns RGB, event-frame, and event-voxel representations in a shared space. The reported results put CM3AE ahead of scratch training, ImageNet-pretrained baselines, and MAE pretrained on the same data on every metric, including larger gains when only 10-20% of downstream labels are available. If the comparison holds, event-vision researchers would no longer need ImageNet weights as a default starting point, and the fusion module itself could become a reusable component for RGB-event models.","feed_headline":"CM3AE tops ImageNet-pretrained models on five event-vision tasks","feed_subtitle":"CM3AE learns from RGB frames, event frames, and voxels, beating ImageNet-pretrained models on five tasks.","key_machinery":"The load-bearing machinery is a dual-branch masked autoencoder in which 75% of patches are masked in both RGB and event frames while half of the unmasked patches are positionally shared between the two modalities; the shared positions prevent the fusion module from trivially recovering masked content from the other modality. On top of this, a multimodal fusion reconstruction module concatenates RGB encoder tokens with event tokens (and, in a second branch, with event-voxel tokens encoded by a small transformer) through a standard Transformer fusion block, then decodes the fused tokens to reconstruct the original RGB image. A multimodal contrastive loss treats RGB, event-frame, and event-voxel views of the same scene as positives and aligns them in a shared normalized space, complementing the local pixel-reconstruction objective with a global correspondence objective. The three losses are summed during pretraining, and the pretrained encoders, decoder, or fusion block are then transferred to downstream models.","core_discovery":"The central claim is that pretraining on paired RGB and event data with a multimodal masked autoencoder yields transferable representations for both unimodal event tasks and RGB-event fusion tasks, and that the pretrained fusion block transfers separately. Concretely, on HARDVS action recognition the model reaches 53.40% Top-1 with RGB+Event input and 52.45% with event-only input, against 50.98% and 49.32% for MAE pretrained on the same REV2M corpus; on EvDET200K detection it reaches 49.0 AP against 47.5; on COESOT tracking it reaches 62.6 SR against 61.0; and on EventVOT it reaches 57.6 SR against 56.8. In few-shot settings with 10% of downstream training data, CM3AE reaches 43.68% Top-1 on RGB+Event action recognition, compared with 33.31% for MAE pretrained on REV2M and 5.63% from scratch. The paper also reports that loading only the pretrained fusion module into downstream RGB-event models improves action recognition Top-1 from 52.40% to 53.40% and tracking SR from 62.0 to 62.6.","pith_inferences":["Editorial inference: the same recipe should transfer to other paired sensor modalities, such as RGB and thermal or RGB and depth, because the fusion-reconstruction and contrastive objectives are not event-specific; only the voxel encoder is.","Editorial inference: the position-sharing rule in the masking step is a candidate load-bearing detail: without it, the fusion module could exploit visible tokens from one modality to reconstruct masked positions of the other, so the masking alignment deserves a dedicated ablation.","Editorial inference: because only one pretraining corpus size is reported, the few-shot results invite a scaling study: if REV2M size is the active ingredient, larger corpora should push the same five benchmarks further, and smaller ones should shrink the gap to MAE."],"forward_implications":["Event-only models can be initialized from a backbone pretrained on paired RGB-event data; on every reported metric this beats both ImageNet-pretrained and REV2M-pretrained MAE initialization.","The pretrained fusion block transfers as a standalone component into RGB-event fusion models and improves them over the same architecture trained from scratch.","With only 10% or 20% of downstream labels, CM3AE initialization produces much larger gains than scratch or MAE-on-REV2M, indicating the pretraining reduces annotation requirements.","A single pretrained model serves both event-frame and event-voxel downstream inputs, so event-only and RGB-event models can share one foundation.","The REV2M corpus is assembled from five public datasets, so the pretraining recipe can be reproduced without new data collection."],"supporting_citations":[{"why":"Supplies the masked autoencoder architecture and reconstruction loss that CM3AE extends to dual-branch multimodal input.","marker":"[19]"},{"why":"Provides the HARDVS action recognition benchmark used for downstream evaluation and the largest share of REV2M pretraining pairs.","marker":"[56]"},{"why":"Supplies the COESOT RGB-event tracking benchmark and the CEUTrack baseline used for downstream tracking evaluation.","marker":"[44]"},{"why":"Provides the Timesformer backbone and protocol for both event-only and RGB-event action recognition experiments.","marker":"[3]"},{"why":"Provides the ViTDet detection model used for event-based object detection evaluation.","marker":"[28]"},{"why":"Provides the ODTrack baseline for event-only visual object tracking.","marker":"[61]"},{"why":"Defines the EvDET200K event object detection benchmark used in the downstream comparison.","marker":"[52]"},{"why":"Defines the EventVOT event tracking benchmark used in the downstream comparison.","marker":"[54]"}],"fun_headline_variants":["CM3AE: one pretraining for RGB, event frames, and voxels","CM3AE beats MAE on five event-vision tasks","CM3AE's cross-modal pretraining lifts five event-vision tasks","Pretrained fusion block alone boosts RGB-event performance","CM3AE: RGB-event pretraining outperforms single-modality MAE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the REV2M pretraining corpus excludes the test splits of HARDVS and COESOT even though it draws 827,694 pairs from HARDVS and 231,277 pairs from COESOT, and those same datasets are then used as downstream benchmarks.","fun_headline_variants_meta":{"raw":{"variants":["CM3AE: one pretraining for RGB, event frames, and voxels","CM3AE beats MAE on five event-vision tasks","CM3AE's cross-modal pretraining lifts five event-vision tasks","Pretrained fusion block alone boosts RGB-event performance","CM3AE: RGB-event pretraining outperforms single-modality MAE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000611,"raw_usage":{"total_tokens":2899,"prompt_tokens":1057,"completion_tokens":1842,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":1750}},"tokens_in":673,"tokens_out":1842,"duration_ms":18127,"temperature":1.0,"reasoning_tokens":1750,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:28:15.061072+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the sequence identifiers of the 827,694 HARDVS pairs and 231,277 COESOT pairs in REV2M against the official HARDVS and COESOT test splits; any overlap means the reported action-recognition and tracking gains could be memorization. Retraining CM3AE and the MAE-on-REV2M baseline on the overlap-free corpus and comparing the gaps would settle the claim.","supporting_citations":[{"cited_title":"Masked autoencoders are scalable vision learners","cited_arxiv_id":null,"evidence_quote":"Supplies the masked autoencoder architecture and reconstruction loss that CM3AE extends to dual-branch multimodal input."},{"cited_title":"Hardvs: Re- visiting human activity recognition with dynamic vision sen- sors","cited_arxiv_id":null,"evidence_quote":"Provides the HARDVS action recognition benchmark used for downstream evaluation and the largest share of REV2M pretraining pairs."},{"cited_title":"Exploring plain vision transformer backbones for object de- tection","cited_arxiv_id":null,"evidence_quote":"Provides the ViTDet detection model used for event-based object detection evaluation."},{"cited_title":"Event stream-based visual object tracking: A high-resolution benchmark dataset and a novel baseline","cited_arxiv_id":null,"evidence_quote":"Defines the EventVOT event tracking benchmark used in the downstream comparison."}],"review_version":1}