Pith. sign in

REVIEW 3 major objections 5 minor 24 references

Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read EventKitchen is presented as the first large-scale stereo event camera dataset of unscripted human cooking, egocentrically recorded with synchronized RGB, depth, and IMU, and labeled for action recognition, object detection, and depth…

desk verdict A genuinely useful real-world egocentric stereo event dataset for daily human activities, held back mainly by a cross-modal annotation pipeline that needs independent validation in the event domain. read the letter →

arxiv 2608.04865 v1 pith:ESUSOEEN submitted 2026-08-05 cs.CV

Chengming Feng , Hesam Araghi , Liming Zheng , Julien Dupeyroux , Xucong Zhang , Jan van Gemert , Nergis Tömen This is my paper · ORCID
classification cs.CV
keywords eventcamerasneuromorphicvisionegocentricstereodepthestimationactionrecognitionobjectdetectionmulti-modaldatasetKitchen
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces EventKitchen, a benchmark dataset claimed to be the first large-scale stereo event camera dataset recorded in a real-world, egocentric, unscripted daily-life setting. Ten participants wore a helmet with stereo event cameras, stereo RGB cameras, a depth camera, and an IMU while cooking in 13 kitchens, yielding 5.5 hours of synchronized streams. Human annotators labeled 10,762 action segments and 13,482 bounding boxes, and the authors provide 297,547 ground-truth depth maps plus calibration matrices. Baseline models for action recognition, object detection, and stereo depth estimation all perform far below their performance on automotive event datasets, which the authors take as evidence that EventKitchen poses genuinely new challenges for neuromorphic vision beyond autonomous driving.

What carries the argument

The carrying mechanism is the multi-modal recording rig and, especially, the cross-modal label projection pipeline. Human annotations are made on RGB frames from the depth camera, where objects are easy to see; each annotated bounding box is lifted into 3D using the aligned depth map, then projected into both event cameras using intrinsic and extrinsic calibration matrices with reported reprojection error below two pixels. Depth maps are likewise projected and rectified into the event stereo frame, so ground truth for two independent event cameras is produced from a single annotation pass in the RGB-depth domain.

What would settle it

Select a random sample of event frames, have independent annotators draw bounding boxes directly on temporally aggregated event images, and compare those boxes with the dataset's projected labels by IoU and center distance; if the disagreement for small or fast objects is much larger than the reported sub-2-pixel reprojection error, the projection-based ground truth is systematically biased.

Watch

Extended reading notes

Core claim

The central claim is that EventKitchen fills a gap: no existing event camera dataset combines real-world capture, stereo event cameras, an egocentric viewpoint, unscripted natural actions, and support for multiple tasks in one benchmark. The dataset consists of 110 sequences from 10 participants across 13 kitchens, with stereo Prophesee Gen4 streams at 1280x720 resolution, synchronized CMOS RGB, depth, and IMU data, plus human annotations of 268 action classes and 12 kitchenware object classes. Baselines show top-1 action accuracy of 24.69% (Swin), object detection AP of 16.2% (YOLOv10), and stereo depth RMSE of 84.91 mm (SE-CFF), which the authors interpret as evidence that the benchmark is complex and that models specialized for automotive or scripted event data do not transfer well.

Load-bearing premise

The load-bearing premise is that labels made on RGB-depth frames and projected into event views through calibration are correct, so if depth noise, calibration error, or timestamp misalignment is larger than reported, every event-domain ground-truth label could be systematically wrong, and the paper gives no independent check of projected labels directly on event frames.

Editorial extensions

If this is right

  • Event-based action recognition can be studied on unscripted, long-tail actions in unseen kitchens; the low top-1 accuracy of strong frame-based baselines quantifies the open gap.
  • Object detection baselines trained for automotive events transfer poorly to egocentric kitchens, and small, fast-moving kitchenware such as forks and spoons is nearly undetected, so new event-specific detectors are needed.
  • Stereo depth estimation is feasible in near-field indoor scenes: SE-CFF reaches RMSE below the ground-truth scatter, while a zero-shot RGB foundation model fails on reconstructed event images.
  • The verb-frequency ranking of EventKitchen closely tracks an established unscripted egocentric cooking benchmark, supporting the paper's claim that the collected actions are natural rather than scripted.
  • The kitchen-level train/test split, with all test kitchens unseen during training, makes EventKitchen a testbed for generalization rather than memorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the projection pipeline is sound, the same annotate-once-in-RGB-depth, project-to-events recipe could be reused to create multi-task event labels for other egocentric settings without requiring annotators to label sparse event streams directly.
  • The especially low detection performance on forks and spoons may partly reflect projection-induced label noise on small, fast, occluded objects rather than detector weakness alone; a direct annotation study on event frames could separate these effects.
  • Because the IMU stream is not yet calibrated to the camera module, as the supplementary material states, a natural extension is to complete that calibration and add egomotion estimation as a fourth benchmarked task.
  • The long-tail action distribution and unseen-kitchen split make EventKitchen a plausible testbed for few-shot learning and domain generalization in event vision, though the paper does not itself run those experiments.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces EventKitchen, a 5.5-hour stereo event camera dataset of unscripted cooking activities, recorded with a wearable helmet equipped with two Prophesee Gen4 event cameras, two CMOS RGB cameras, an Intel RealSense depth camera, and an IMU. Ten participants were recorded in 13 kitchens. Human annotations of 10,762 action segments and 13,482 bounding boxes are made on D-RGB frames and projected into the left and right event camera views via calibration matrices; depth maps are also rectified into event camera coordinates. The authors train baseline models for event-based action recognition (TSM, Swin), object detection (YOLOv10, RVT, EvRT-DETR), and stereo depth estimation (SE-CFF, FoundationStereo), reporting low performance as evidence of task difficulty, and claim that EventKitchen is the first large-scale, real-world, egocentric, stereo event camera benchmark supporting multiple tasks.

Significance. If the ground truth is reliable, EventKitchen is a valuable community resource: it fills a genuine gap in human-centric daily-life event vision, uses real unscripted activities, combines stereo events with RGB-D and IMU data, provides annotations for three tasks, and includes reproducible baseline configurations and a public toolkit. The low baseline numbers are informative for the field. However, the benchmark's value hinges on the accuracy of projecting RGB-D annotations and depth maps into the event domain; without independent event-domain validation, the risk of systematic label bias is substantial. The manuscript is strongest in its detailed recording setup, transparent participant statistics, and the public release of data and tools.

major comments (3)
  1. [Section 3.3, Fig. 3] The central load-bearing step is the projection of bounding boxes and depth maps from the D-RGB/depth domain into the left and right event camera views, but this projection is never validated on event-domain data. The reported <2 px reprojection errors come from checkerboard calibration, not from annotated objects, and do not bound the full-pipeline error: depth noise (<2% at 2 m, roughly 40 mm), axis-aligned 3D boxes that inevitably include background or occluding points, and temporal misalignment can systematically enlarge or shift projected boxes. This is especially relevant for the fork and spoon classes, which obtain near-zero AP and could reflect label bias rather than genuine task difficulty. The authors should provide an independent validation, e.g., human labels collected directly on event reconstructions for a subset of frames, and report agreement against the projected boxes.
  2. [Section 3.1, Section 3.4, Supplementary Section 1] The manuscript contains a material contradiction about the IMU. The main paper lists '5.5 hours of 6-axis IMU data at 200 fps' and shows an IMU column in Fig. 1, while the supplementary material states that the IMU is physically present but 'excluded from the main paper because its calibration relative to the camera module has not been fully validated' and the supplementary Fig. 1 describes a 9-axis IMU. As written, a user cannot know whether IMU data are part of the released dataset or whether they are synchronized and calibrated. This inconsistency must be resolved in the main text and the released metadata.
  3. [Section 3.1, 'Synchronization'] Temporal alignment between the 15 fps D-RGB/depth stream and the asynchronous event streams is asserted via ROS timestamps but is not validated. A few milliseconds of offset is significant for the 50 ms detection windows used in Section 4.2 and for fast actions such as stirring, so the authors should report a concrete synchronization validation, for example by cross-correlating motion onset between D-RGB frames and event rates, or by showing that projected boxes track moving objects in event reconstructions with bounded lag.
minor comments (5)
  1. [Section 3.4 vs. Table 1] The number of depth maps is inconsistent: Section 3.4 reports 297,547 raw depth maps while Table 1 reports 297,457. Please correct the discrepancy.
  2. [Fig. 5 caption] The caption states that the figures 'demonstrate our projection pipeline is not biased' because the left and right cameras have similar numbers of bounding boxes per class, but equal class counts do not demonstrate geometric accuracy; this wording should be softened or replaced with a geometric validation.
  3. [Section 4.2] The metric AP05 (IoU threshold 0.05) is used without definition or justification; please define it and explain its role, since it is not a standard COCO metric.
  4. [Supplementary Section 3] The implementation list contains the typo 'FundationStereo' for 'FoundationStereo'; please fix this in the final version.
  5. [Table 3] The comparison of verb frequency rankings with EPIC-KITCHENS is somewhat opaque because 'Rinse' and 'Soap' are counted separately in EventKitchen but merged into 'Wash' for EPIC-KITCHENS; a short clarifying note in the table caption or text would help.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: EventKitchen is a self-contained empirical dataset paper with external baselines and no fitted-input-as-prediction chain.

full rationale

EventKitchen is a dataset paper whose contributions are empirical: new recordings, human annotations, calibration, and baseline evaluations. The claimed novelty (the only real-world egocentric stereo event-camera dataset supporting multiple tasks) is supported by a literature comparison in Table 1 against independent prior datasets, not by a derivation from its own assumptions. The annotation pipeline projects human labels from D-RGB/depth into event views, but this is a data-generation procedure, not a prediction of the dataset's own fitted parameters; the baselines are external public models (TSM, Swin, YOLOv10, RVT, EvRT-DETR, SE-CFF, FoundationStereo) pretrained on independent data such as Kinetics, COCO, and 1Mpx. No equation is defined in terms of a result it is claimed to predict, and no load-bearing claim rests on the authors' own prior work being treated as an external fact. The only self-citation (ref. [45], Paredes-Vallés et al., which includes author Dupeyroux) appears in a routine list of robotics uses of event cameras and is not load-bearing. Concerns about label projection accuracy or sub-2-pixel reprojection error are validity risks, not circularity: they do not make any result equivalent to its input by construction. The paper is self-contained against external benchmarks, so the correct finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The dataset's validity rests on calibration, synchronization, and depth-quality assumptions rather than on mathematical axioms; these are practical domain assumptions typical of dataset construction.

assumptions (3)
  • domain assumption Calibration matrices with sub-2-pixel reprojection error are accurate enough for projecting annotations across modalities.
    Invoked in Section 3.3 and supplement Table 1.
  • domain assumption RealSense depth maps are reliable ground truth for near-field kitchen scenes (<2% error at 2 m), and the projection into event camera FoV does not introduce significant bias.
    Section 3.3 depth map rectification pipeline.
  • domain assumption ROS timestamps provide sufficient synchronization across sensors, including asynchronous event cameras, for temporal alignment of annotations.
    Section 3.1 Synchronization paragraph.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen." pith.science (2026). https://pith.science/paper/ESUSOEEN

@misc{pith2026260804865,
  author       = {Pith},
  title        = {Pith review of: Cooking beyond Frames: A Stereo Event Camera Dataset in the Kitchen},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ESUSOEEN}},
  note         = {Machine review of arXiv:2608.04865}
}
read the original abstract

Event cameras, also known as neuromorphic cameras, have gained significant attention in recent years due to their high temporal resolution, high dynamic range, and low power consumption. While many studies and datasets in neuromorphic vision have focused on automotive and drone applications, human-centric daily-life scenarios remain largely underrepresented, despite their importance for developing and benchmarking event-based perception systems. Moreover, the few existing event-based human activity datasets are typically recorded with scripted human actions, limiting their ability to capture natural human behaviors. In this paper, we introduce EventKitchen, a large-scale stereo event camera benchmark dataset of human cooking activities in the kitchen. EventKitchen is egocentrically collected from 10 participants in 13 diverse kitchens, where the participants wear a helmet with multiple sensors and naturally perform cooking activities, without any scripted actions. EventKitchen comprises 5.5 hours of stereo event recordings with synchronized RGB, depth, and IMU data. We provide human annotations for 10,762 action segments and 13,482 bounding boxes. We train baseline models on EventKitchen to perform multiple event-based tasks, including action recognition, object detection, and stereo depth estimation. By capturing natural, real-world human activities, EventKitchen establishes a challenging benchmark for neuromorphic vision beyond autonomous driving.

Figures

Figures reproduced from arXiv: 2608.04865 by the authors.

Figure 1
Figure 1. (a) Data collection device: The device is a wearable bicycle helmet with mounted sensors including two event cameras, two CMOS RGB cameras, and one Intel RealSense depth camera to allow multi-modal sensing; (b) Raw data: We show seven example raw data streams collected with the device in three out of 13 diverse kitchens, columns ‘IMU’, ‘D-RGB’, and ‘Depth’ are from Intel RealSense, ‘Left RGB’ and ‘Right RGB’ from tw… view at source ↗
Figure 2
Figure 2. Top: Rectified and annotated frames from four different time points t0, ..., t3. Frames depict the ground truth depth maps and corresponding events aggregated from the left and right event cameras, along with human-annotated object bounding boxes. RGB references highlight the difficulty of annotating on events directly; Middle: Video streams from the depth camera, and left and right event cameras. Event colors (red,… view at source ↗
Figure 3
Figure 3. Bounding box annotation pipeline: 1. Obtain human-annotated bounding boxes on the RGB frame; 2. Construct 3D points of each bounding box based on its synchronized depth map; 3. Project 3D points to the left and right event camera using intrinsic and extrinsic matrices. Depth map rectification pipeline: 1. Construct 3D points for the depth map; 2. Project all 3D points to the target camera domain (left or right event… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Left: Distribution of the 32 annotated action verbs in EventKitchen; Right: Distribution of the 268 annotated actions in EventKitchen. The two distributions demonstrate the variance of action verbs and action classes in EventKitchen, and the realistic long tails presen…
Figure 5
Figure 5. Figure 5: Left: Distribution of the 12 object classes in terms of number of samples in EventKitchen; Middle: Distribution of the central coordinate of all bounding boxes; Right: Distribution of the size of all bounding boxes. These figures demonstrate our projection pipeline is …
Figure 6
Figure 6. Figure 6: Left: Depth histogram from all 10,800 samples is in the range from 200 mm to 1500 mm; Right: Mean and standard deviation of depth per session. Depth information is within an expected range for kitchen activities where objects are typically about 700 mm away [PITH_FULL…
Figure 4
Figure 4. Figure 4: Class imbalance remains a significant challenge for baseline models in ob [PITH_FULL_IMAGE:figures/full_fig_p014_4.png]
Figure 1
Figure 1. Figure 1: Detailed illustration of our data collection device. [PITH_FULL_IMAGE:figures/full_fig_p021_1.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 12 canonical work pages

  1. [1]

    In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

    Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 6299–6308 (2017)

    2017

  2. [2]

    In: Proceedings of the European conference on computer vision (ECCV)

    Damen, D., Doughty, H., Farinella, G.M., Fidler, S., Furnari, A., Kazakos, E., Moltisanti, D., Munro, J., Perrett, T., Price, W., et al.: Scaling egocentric vision: The epic-kitchens dataset. In: Proceedings of the European conference on computer vision (ECCV). pp. 720–736 (2018)

    2018

  3. [3]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Gehrig, M., Scaramuzza, D.: Recurrent vision transformers for object detection with event cameras. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13884–13893 (2023)

    2023

  4. [4]

    He,K.,Zhang,X.,Ren,S.,Sun,J.:Deepresiduallearningforimagerecognition.In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770–778 (2016)

    2016

  5. [5]

    arXiv preprint arXiv:1412.6980 (2014) 26 C.Feng et al

    Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014) 26 C.Feng et al

    arXiv 2014

  6. [6]

    (Jul 2021).https://doi.org/10.5281/ zenodo.5079802,https://doi.org/10.5281/zenodo.5079802, Documentation available under https://tonic.readthedocs.io

    Lenz, G., Chaney, K., Shrestha, S.B., Oubari, O., Picaud, S., Zarrella, G.: Tonic: event-based datasets and transformations. (Jul 2021).https://doi.org/10.5281/ zenodo.5079802,https://doi.org/10.5281/zenodo.5079802, Documentation available under https://tonic.readthedocs.io

    doi:10.5281/zenodo.5079802 2021

  7. [7]

    In: Proceedings of the IEEE/CVF international conference on computer vision

    Lin, J., Gan, C., Han, S.: Tsm: Temporal shift module for efficient video under- standing. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 7083–7093 (2019)

    2019

  8. [8]

    In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13

    Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer vision– ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13. pp. 740–755. Springer (2014)

    2014

Show all 24 references
  1. [9]

    BMVC (2018)

    Liu, M., Delbruck, T.: Adaptive time-slice block-matching optical flow algorithm for dynamic vision sensors. BMVC (2018)

    2018

  2. [10]

    In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition

    Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin trans- former. In: Proceedings of the IEEE/CVF conference on computer vision and pat- tern recognition. pp. 3202–3211 (2022)

    2022

  3. [11]

    arXiv preprint arXiv:1711.05101 (2017)

    Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

    2017 arXiv

  4. [12]

    IEEE transactions on pattern analysis and machine intelligence44(10), 6890–6909 (2021)

    Mostafavi, M., Nam, Y., Choi, J., Yoon, K.J.: E2sri: Learning to super-resolve intensity images from events. IEEE transactions on pattern analysis and machine intelligence44(10), 6890–6909 (2021)

    2021

  5. [13]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Nam, Y., Mostafavi, M., Yoon, K.J., Choi, J.: Stereo depth from events cameras: Concentrate and focus on the future. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 6114–6123 (2022)

    2022

  6. [14]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Plizzari, C., Planamente, M., Goletto, G., Cannici, M., Gusso, E., Matteucci, M., Caputo, B.: E2 (go) motion: Motion augmented event stream for egocentric action recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 19935–19947 (2022)

    2022

  7. [15]

    IEEE Conf

    Rebecq, H., Ranftl, R., Koltun, V., Scaramuzza, D.: Events-to-video: Bringing modern computer vision to event cameras. IEEE Conf. Comput. Vis. Pattern Recog. (CVPR) (2019)

    2019

  8. [16]

    arXiv preprint arXiv:1609.04747 (2016)

    Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016)

    2016 arXiv

  9. [17]

    In: Artificial intelligence and machine learning for multi- domain operations applications

    Smith, L.N., Topin, N.: Super-convergence: Very fast training of neural networks using large learning rates. In: Artificial intelligence and machine learning for multi- domain operations applications. vol. 11006, pp. 369–386. SPIE (2019)

    2019

  10. [18]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Torbunov, D., Ren, Y., Ghose, A., Dim, O., Cui, Y.: Evrt-detr: Latent space adap- tation of image detectors for event-based vision. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 9812–9821 (2025)

    2025

  11. [19]

    Advances in Neural Information Processing Systems 37, 107984–108011 (2025)

    Wang, A., Chen, H., Liu, L., Chen, K., Lin, Z., Han, J., et al.: Yolov10: Real-time end-to-end object detection. Advances in Neural Information Processing Systems 37, 107984–108011 (2025)

    2025

  12. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Wang, L., Ho, Y.S., Yoon, K.J., et al.: Event-based high dynamic range image and very high frame rate video generation using conditional generative adversarial networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10081–10090 (2019)

    2019

  13. [21]

    Wang, Q., Xu, Z., Lin, Y., Ye, J., Li, H., Zhu, G., Ali Shah, S.A., Bennamoun, M., Zhang, L.: Dailydvs-200: A comprehensive benchmark dataset for event-based ac- tionrecognition.In:EuropeanConferenceonComputerVision.pp.55–72.Springer (2024) Cooking beyond Frames: A Stereo Even...

    2024

  14. [22]

    In: Proceedings of the AAAI conference on artificial intelligence

    Wang, X., Wu, Z., Jiang, B., Bao, Z., Zhu, L., Li, G., Wang, Y., Tian, Y.: Hardvs: Revisiting human activity recognition with dynamic vision sensors. In: Proceedings of the AAAI conference on artificial intelligence. vol. 38, pp. 5615–5623 (2024)

    2024

  15. [23]

    CVPR (2025)

    Wen, B., Trepte, M., Aribido, J., Kautz, J., Gallo, O., Birchfield, S.: Foundation- stereo: Zero-shot stereo matching. CVPR (2025)

    2025

  16. [24]

    Zhu, A.Z., Yuan, L., Chaney, K., Daniilidis, K.: Unsupervised event-based learning of opticalflow,depth, and egomotion.In: Proceedingsof theIEEE/CVF conference on computer vision and pattern recognition. pp. 989–997 (2019)

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.