Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Masked autoencoding of raw event points achieves state-of-the-art action recognition on DVS benchmarks while preserving temporal structure.

desk verdict First to apply masked autoencoding to raw event streams, but the SOTA claim is confounded by ShapeNet pretraining and the empirical support is thinner than the tables suggest. read the letter →

arxiv 2501.01040 v1 pith:4KGHD6M2 submitted 2025-01-02 cs.CV

Jingkai Sun , Qiang Zhang , Jiaxu Wang , Jiahang Cao , Renjing Xu This is my paper · ORCID
classification cs.CV
keywords eventcameraactionrecognitionmaskedautoencoderself-supervisedlearningpointcloudinliermodeldynamicvisionsensortransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that self-supervised masked reconstruction, previously successful for images and point clouds, works on raw event streams from dynamic vision sensors. It treats events as point clouds and pretrains a transformer by masking event patches and reconstructing them, using a plane-fitting inlier model to select clean patch centers. If correct, event-camera action recognition can skip lossy frame-based conversions and still beat prior methods. The authors report top accuracy on DVS128-Gesture (98.54 percent, 10 classes), DVS Action (93.9 percent), and SL-Animals-DVS S3 (88.23 percent), while lagging slightly on noise-heavy S4 (87.46 versus EvT's 88.12).

What carries the argument

Event patch generation via inlier filtering: each candidate patch center is accepted only if its K-nearest neighbors fit a plane t = ax + by + c with average residual below a threshold H. The chosen centers define event patches; PointNet embeds each patch into a token, a standard transformer encoder sees only visible tokens, and a decoder reconstructs masked patches under Chamfer Distance loss. Pretraining starts from ShapeNet point-cloud weights, then continues on event data, followed by fine-tuning for classification.

What would settle it

Train the same model on a synthetic event dataset with controlled non-linear acceleration while varying the threshold H, and compare against a random-sampling baseline; if random sampling matches or beats the inlier method as acceleration grows, the planarity assumption is the binding limit. Alternatively, measure classification accuracy on the SL-Animals-DVS S4 noisy split against a version with no plane filtering; if removing the filter does not lower accuracy, the denoising step is not carrying the claimed benefit.

Watch

Extended reading notes

Core claim

The central claim is that masked autoencoding applied directly to raw event points—with an event inlier model that rejects noise when selecting patch centers—learns representations that outperform existing frame-based and point-based action recognition models on several public benchmarks. The paper shows that Farthest Point Sampling, standard for point clouds, degrades on event data because the time axis does not encode shape and because noise biases FPS toward outliers. Replacing FPS with least-squares plane fitting over local neighborhoods selects centers whose groups lie on near-planar structures in (x,y,t), lowering pretraining Chamfer Distance by roughly fourfold and improving downstream classification. The method's weaker result on the noisier SL-Animals-DVS S4 split is attributed to limits of the plane-fitting denoiser.

Load-bearing premise

The plane-fitting center selection assumes that, within a short time window, object motion is locally linear so nearby events lie on a plane in (x,y,t); when motion is highly non-linear or noise is severe, this assumption filters out informative events.

Editorial extensions

If this is right

  • Masked modeling is shown to be effective for raw event stream data, not just images and point clouds.
  • Pre-training on ShapeNet point clouds transfers to event data, reducing reconstruction loss by 2.3 × 10^-4 compared to training from scratch.
  • The inlier-based patch center sampler cuts pretraining Chamfer Distance roughly fourfold versus FPS and also improves on random sampling.
  • A single masked-autoencoder backbone may later unify event data, point clouds, and images for multi-modal learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the local-planarity assumption holds, the same patch selection could be adapted to event-based optical flow or low-latency tracking, where denoising raw events matters more than classification accuracy.
  • The S4 shortfall suggests a testable boundary: on datasets with fast non-linear motion or strong sensor noise, an adaptive threshold that scales with local event density might outperform the fixed hyperparameter H.
  • The 0.5-second sliding window choice implies a latency ceiling; shrinking the window would trade accuracy for responsiveness in real-time robotics applications.
  • A direct comparison against a random-sampling MAE without plane fitting would isolate how much of the gain comes from denoising versus from masked pretraining itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Event Masked Autoencoder (Event-MAE), which applies Point-MAE-style masked autoencoding to DVS event streams treated as 3D point clouds (x, y, t). The main novelties are an event patch generation algorithm that uses plane-fitting inlier selection (Eqs. 2-4) with KNN grouping and an inlier threshold H, and a PointNet-based patch embedding for transformer models. The model is pre-trained on ShapeNet and then on event datasets, and fine-tuned for classification on DVS128-Gesture, SL-Animals-DVS, and DVS Action. The paper reports state-of-the-art accuracy on several benchmarks (Table I: 98.54/97.75; Table II S3: 88.23; Table III: 93.9) and provides ablations of the threshold and sampling method.

Significance. If the reported results are reproducible and the comparison is fair, the paper demonstrates that masked autoencoding on raw event point clouds is viable and can outperform frame-based and point-based baselines, opening a promising direction for unified point/event pre-training. The paper is well aligned with current MAE literature, and the ablation of patch generation methods provides useful evidence that FPS is poorly suited to event data. However, the strength of the empirical claim is substantially weakened by an internal contradiction on DVS Action, a confounded pre-training comparison, test-set hyperparameter selection, and a lack of variance estimates, so the central 'state-of-the-art' claim is not yet established.

major comments (4)
  1. [Section III-B / Table III] The text states: 'DVS Action is a dataset smaller than DVS128 Gesture Dataset. So we do not fine-tune our model on this dataset,' yet Table III reports 'Our 93.9' for DVS Action. This is internally inconsistent and makes the DVS Action result uninterpretable. The authors must state exactly how the 93.9% figure was obtained (fine-tuning protocol, which pretrained checkpoint was used, evaluation setup) or remove the result from the comparison.
  2. [Section III-A / Tables I-III] The model is initialized from a ShapeNet-pretrained Point-MAE checkpoint, and the authors report that this reduces the loss by 2.3e-4 compared with event-only training. The baselines in Tables I-III (EvT, PointNet++, TORE, etc.) are trained on event data without such external pretraining. Because the claimed contribution is the event-specific patch generation and masked modeling, the reported gains (0.14-0.78 percentage points over EvT) are confounded by transfer from point-cloud data. To support the SOTA claim, provide an ablation with the same backbone trained from scratch or with event-only pretraining, and/or pretrain the baselines under the same initialization protocol.
  3. [Section III-C / Table IV] The inlier threshold H is selected based on DVS128-Gesture classification accuracy (the best value, H=0.85, is identified in Table IV), and the same dataset's accuracy is then reported as the headline result in Table I. This is selection on the test set; without a separated validation split or repeated cross-validation, the reported accuracy is optimistically biased. Please report validation-based selection or show that the result is stable across H values and across multiple runs.
  4. [Tables I-III (overall)] The reported margins over the strongest baseline EvT are 0.14, 0.22, and 0.78 percentage points on DVS128-Gesture 10/11 classes and SL-Animals S3, and on SL-Animals S4 the proposed method (87.46) is below EvT (88.12). No error bars, standard deviations, or numbers of seeds are reported anywhere in the tables. With these margins, the claim that the method 'outperforms the state-of-the-art' is not statistically supported; add repeated-run statistics or at least report the number of runs for each result.
minor comments (6)
  1. [Section II-A] The normalization equation is written as 't = tn−t0 / tmax−t0', which is ambiguous; it should be t = (tn−t0)/(tmax−t0).
  2. [Equation (4)] The definition of erj is unclear: the expression uses a bare 'P' symbol without a defined summation, and the division by n is not visually attached to the sum. Please rewrite as a standard average, e.g., erj = (1/n) * sum_i (Delta t_i - Delta t̂_i).
  3. [Section II-B] The phrase 'K-Nearest Neighborhood' should be 'K-Nearest Neighbor', and the notation P = KNN(E, C_I) is confusing because P is later used for patches; please define all variables consistently.
  4. [Section III-A / Table IV] The value of the inlier threshold H used for the main results is not stated in Section III-A; it appears only in the ablation. Please report the default H used for Tables I-III.
  5. [Figure 5 caption] The caption says 'we contact two event data streams'; this is likely a typo for 'concatenate'. Please correct the wording.
  6. [Reference [20]] The position embedding is said to follow [20], which is the BERT paper; the relevant reference for point-cloud patch position embedding is more likely Point-MAE [11] or Point-BERT [14]. Please cite the appropriate source.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical masked-autoencoder study, and its benchmark claims are not derived from their own inputs by construction.

full rationale

This paper makes no first-principles derivation; its central claims are experimental. The event patch generation uses standard KNN plane fitting (Eqs. 1-4) and a threshold H chosen as a hyperparameter; the masked autoencoder uses Point-Net embedding and Chamfer Distance reconstruction loss. None of these steps defines the reported accuracy in terms of the method's own assumptions or fitted parameters by construction. The threshold H is selected by comparing downstream accuracy in Table IV, which is a hyperparameter-selection procedure and a possible overfitting concern, but it is not a circular derivation because the reported accuracy is not algebraically or statistically forced by the threshold choice. The ShapeNet pretraining confound noted in Sec. III-A is a fairness/correctness issue regarding comparison with baselines, not circularity. The self-citations in the introduction ([2], [3]) are not load-bearing for the main claims. Overall, the benchmark evaluations are externally referenced, and the paper is self-contained as an empirical study rather than a derivation chain.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The empirical claims depend on several hand-set hyperparameters, most notably the inlier threshold H tuned on the evaluation benchmark. The method also assumes that events behave as 3D points, that local event trajectories are planar, that ShapeNet point-cloud weights transfer to event data, and that Chamfer distance is a good reconstruction objective. No new physical entities are introduced.

free parameters (6)
  • Inlier threshold H = 0.85
    Chosen via ablation on DVS128-Gesture (Table IV); controls how many patch centers are kept and directly affects pretraining loss and downstream accuracy.
  • KNN group size k = 32
    Number of events per patch; set to 32 with no sensitivity analysis reported.
  • Number of patches m = 64
    Number of event patches per sliding window; fixed without ablation.
  • Masking ratio alpha = 0.8 (80%)
    Used in Figure 1 and the pipeline; no reported ablation on alpha in the experiments.
  • Sliding window length and step = 0.5 s and 0.25 s
    Used to sample event streams into windows; chosen by hand and not ablated.
  • Input events per window N = 1024
    Number of events sampled per window, inherited from Point-MAE conventions; not justified for event data.
assumptions (5)
  • domain assumption Events can be treated as 3D points (x,y,t) and point cloud networks can consume them directly.
    Section II-A converts event data into 3D point-wise data; this is the basis of all experiments.
  • ad hoc to paper Local event patches lie approximately on a plane (linear motion over short time delta).
    Section II-B: 'we assume the movement of the object is linear in δt, δt→0'; this is the core denoising premise and is likely violated in strong noise, as the S4 result suggests.
  • domain assumption ShapeNet point-cloud pretrained weights transfer useful local geometry to event streams.
    Section III-A states that a ShapeNet-pretrained model is better than training from scratch, but provides no controlled study or theoretical justification.
  • domain assumption Chamfer Distance is an appropriate reconstruction objective for masked event patches.
    Section II-C uses Chamfer Distance as the pretraining loss without comparing alternative reconstruction losses.
  • domain assumption Benchmark comparisons against published numbers are apples-to-apples.
    Tables I III compare against prior published accuracies, but the paper does not describe evaluation protocol details for each dataset.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras." pith.science (2026). https://pith.science/paper/4KGHD6M2

@misc{pith2026250101040,
  author       = {Pith},
  title        = {Pith review of: Event Masked Autoencoder: Point-wise Action Recognition with Event-Based Cameras},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4KGHD6M2}},
  note         = {Machine review of arXiv:2501.01040}
}
read the original abstract

Dynamic vision sensors (DVS) are bio-inspired devices that capture visual information in the form of asynchronous events, which encode changes in pixel intensity with high temporal resolution and low latency. These events provide rich motion cues that can be exploited for various computer vision tasks, such as action recognition. However, most existing DVS-based action recognition methods lose temporal information during data transformation or suffer from noise and outliers caused by sensor imperfections or environmental factors. To address these challenges, we propose a novel framework that preserves and exploits the spatiotemporal structure of event data for action recognition. Our framework consists of two main components: 1) a point-wise event masked autoencoder (MAE) that learns a compact and discriminative representation of event patches by reconstructing them from masked raw event camera points data; 2) an improved event points patch generation algorithm that leverages an event data inlier model and point-wise data augmentation techniques to enhance the quality and diversity of event points patches. To the best of our knowledge, our approach introduces the pre-train method into event camera raw points data for the first time, and we propose a novel event points patch embedding to utilize transformer-based models on event cameras.

Figures

Figures reproduced from arXiv: 2501.01040 by the authors.

Figure 1
Figure 1. Reconstruction result of right arm rotation clock [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Pipeline of our Event Masked Autoencoder. In the [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Reconstruction examples on DVS128-Gesture test set. [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: We show three different thresholds. For visualization, [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 4
Figure 4. Figure 4: A comparison of three algorithms for generating event [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EV-Flying: an Event-based Dataset for In-The-Wild Recognition of Flying Objects

    cs.CV 2025-06 conditional novelty 6.0 of 10

    EV-Flying is a hand-annotated event-camera dataset of birds, insects, and drones, with a PointNet++ benchmark reaching about 72% single-chunk and 92% full-track accuracy.

Reference graph

Works this paper leans on

26 extracted references · 16 canonical work pages · cited by 1 Pith paper

  1. [1]

    Event- based vision: A survey,

    G. Gallego, T. Delbr ¨uck, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, S. Leutenegger, A. J. Davison, J. Conradt, K. Daniilidis, et al., “Event- based vision: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 1, pp. 154–180, 2020

    2020

  2. [2]

    Fully spiking neural network for legged robots,

    X. Jiang, Q. Zhang, J. Sun, J. Cao, J. Ma, and R. Xu, “Fully spiking neural network for legged robots,” arXiv preprint arXiv:2310.05022 , 2023

    arXiv 2023

  3. [3]

    Evggs: A collaborative learning framework for event-based generalizable gaussian splatting,

    J. Wang, J. He, Z. Zhang, M. Sun, J. Sun, and R. Xu, “Evggs: A collaborative learning framework for event-based generalizable gaussian splatting,” arXiv preprint arXiv:2405.14959 , 2024

    arXiv 2024

  4. [4]

    Human action recognition and prediction: A survey,

    Y . Kong and Y . Fu, “Human action recognition and prediction: A survey,” International Journal of Computer Vision , vol. 130, no. 5, pp. 1366–1401, 2022

    2022

  5. [5]

    St-evnet: Hierarchical spatial and temporal feature learning on space-time event clouds,

    Q. Wang, Y . Zhang, J. Yuan, and Y . Lu, “St-evnet: Hierarchical spatial and temporal feature learning on space-time event clouds,” in Proc. Adv. Neural Inf. Process. Syst.(NeurlIPS) , 2020

    work page 2020

  6. [6]

    Space-time event clouds for gesture recognition: From rgb cameras to event cameras,

    ——, “Space-time event clouds for gesture recognition: From rgb cameras to event cameras,” in 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) . IEEE, 2019, pp. 1826– 1835

    work page 2019

  7. [7]

    E2 (go) motion: Motion augmented event stream for egocentric action recognition,

    C. Plizzari, M. Planamente, G. Goletto, M. Cannici, E. Gusso, M. Matteucci, and B. Caputo, “E2 (go) motion: Motion augmented event stream for egocentric action recognition,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2022, pp. 19 935–19 947

    work page 2022

  8. [8]

    Speed invariant time surface for learning to detect corner points with event-based cameras,

    J. Manderscheid, A. Sironi, N. Bourdis, D. Migliore, and V . Lepetit, “Speed invariant time surface for learning to detect corner points with event-based cameras,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 10 245–10 254

    2019

Show all 26 references
  1. [9]

    A low power, fully event-based gesture recognition system,

    A. Amir, B. Taba, D. Berg, T. Melano, J. McKinstry, C. Di Nolfo, T. Nayak, A. Andreopoulos, G. Garreau, M. Mendoza, et al., “A low power, fully event-based gesture recognition system,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp....

    2017

  2. [10]

    Event transformer. a sparse-aware solution for efficient event data processing,

    A. Sabater, L. Montesano, and A. C. Murillo, “Event transformer. a sparse-aware solution for efficient event data processing,” in Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 2677–2686

    2022

  3. [11]

    Masked autoencoders for point cloud self-supervised learning,

    Y . Pang, W. Wang, F. E. Tay, W. Liu, Y . Tian, and L. Yuan, “Masked autoencoders for point cloud self-supervised learning,” in Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part II . Springer, 2022, pp. 604–621

    2022

  4. [12]

    Reducing the sim-to-real gap for event cameras,

    T. Stoffregen, C. Scheerlinck, D. Scaramuzza, T. Drummond, N. Barnes, L. Kleeman, and R. Mahony, “Reducing the sim-to-real gap for event cameras,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVII 16. Springer, 202...

    2020

  5. [13]

    Self-supervised learning of inlier events for event-based optical flow,

    J. Nagata and Y . Aoki, “Self-supervised learning of inlier events for event-based optical flow,” 2022

    2022

  6. [14]

    Point-bert: Pre-training 3d point cloud transformers with masked point modeling,

    X. Yu, L. Tang, Y . Rao, T. Huang, J. Zhou, and J. Lu, “Point-bert: Pre-training 3d point cloud transformers with masked point modeling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 19 313–19 322

    2022

  7. [15]

    Time-ordered recent event (tore) volumes for event cameras,

    R. W. Baldwin, R. Liu, M. Almatrafi, V . Asari, and K. Hirakawa, “Time-ordered recent event (tore) volumes for event cameras,” IEEE Transactions on Pattern Analysis and Machine Intelligence , vol. 45, no. 2, pp. 2519–2532, 2022

    2022

  8. [16]

    Temporal binary representation for event-based action recognition,

    S. U. Innocenti, F. Becattini, F. Pernici, and A. Del Bimbo, “Temporal binary representation for event-based action recognition,” in 2020 25th International Conference on Pattern Recognition (ICPR). IEEE, 2021, pp. 10 426–10 432

    2020

  9. [17]

    Introduction and analysis of an event-based sign lan- guage dataset,

    A. Vasudevan, P. Negri, B. Linares-Barranco, and T. Serrano- Gotarredona, “Introduction and analysis of an event-based sign lan- guage dataset,” in 2020 15th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2020) . IEEE, 2020, pp. 675–682

    2020

  10. [18]

    Synaptic plasticity dynamics for deep continuous local learning (decolle),

    J. Kaiser, H. Mostafa, and E. Neftci, “Synaptic plasticity dynamics for deep continuous local learning (decolle),” Frontiers in Neuroscience, vol. 14, p. 424, 2020

    2020

  11. [19]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proceedings of the IEEE/CVF international conference on computer vision , 2021, pp. 10 012–10 022

    2021

  12. [20]

    Bert: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805, 2018

    2018 arXiv

  13. [21]

    Event-based action recognition using motion information and spiking neural networks

    Q. Liu, D. Xing, H. Tang, D. Ma, and G. Pan, “Event-based action recognition using motion information and spiking neural networks.” in IJCAI, 2021, pp. 1743–1749

    2021

  14. [22]

    An event-driven categorization model for aer image sensors using multispike encoding and learning,

    R. Xiao, H. Tang, Y . Ma, R. Yan, and G. Orchard, “An event-driven categorization model for aer image sensors using multispike encoding and learning,” IEEE transactions on neural networks and learning systems, vol. 31, no. 9, pp. 3649–3657, 2019

    2019

  15. [23]

    Action recognition and benchmark using event cameras,

    Y . Gao, J. Lu, S. Li, N. Ma, S. Du, Y . Li, and Q. Dai, “Action recognition and benchmark using event cameras,” IEEE Transactions on Pattern Analysis and Machine Intelligence , 2023

    2023

  16. [24]

    Pointnet: Deep learning on point sets for 3d classification and segmentation,

    C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2017, pp. 652–660

    2017

  17. [25]

    Shapenet: An information-rich 3d model repository,

    A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. , “Shapenet: An information-rich 3d model repository,” arXiv preprint arXiv:1512.03012, 2015

    2015 arXiv

  18. [26]

    Sl-animals-dvs: event-driven sign language animals dataset,

    A. Vasudevan, P. Negri, C. Di Ielsi, B. Linares-Barranco, and T. Serrano-Gotarredona, “Sl-animals-dvs: event-driven sign language animals dataset,” Pattern Analysis and Applications , pp. 1–16, 2021

    2021

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.