pith. sign in

arxiv: 2604.22476 · v1 · submitted 2026-04-24 · 💻 cs.CV · cs.LG

All Eyes on the Workflow: Automated and Efficient Event Discovery from Video Streams

Marco Pegoraro , Jonas Seng , Dustin Heller , Wil M.P. van der Aalst , Kristian Kersting This is my paper

Pith reviewed 2026-05-08 12:41 UTC · model grok-4.3

classification 💻 cs.CV cs.LG
keywords event discoveryvideo streamsprocess miningimage embeddingstemporal segmentationfew-shot classificationworkflow analysis
0
0 comments X

The pith

SnapLog converts video streams into timestamped process event logs using embeddings and similarity matrices.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SnapLog to bridge video data and process mining by automatically extracting interpretable events from footage. Frames are turned into feature vectors with image embeddings, then segmented by computing similarity matrices across time to find boundaries. A few-shot classification step then labels the resulting segments as specific events with timestamps. This matters because many real workflows exist only as video recordings that current process analysis tools cannot use directly.

Core claim

SnapLog produces logs that accurately reflect the process in the videos by converting frames to feature vectors using image embeddings, performing temporal segmentation through frame-wise similarity matrices, and applying generalized few-shot classification to assign labels to the video segments.

What carries the argument

The SnapLog pipeline: pre-trained image embeddings create per-frame vectors, frame-wise similarity matrices identify temporal cuts between events, and generalized few-shot classification labels the resulting segments.

If this is right

  • Process mining techniques become applicable to video recordings of workflows without manual event logging.
  • Event discovery from videos is automated, reducing the effort needed to create usable process data.
  • The resulting logs support standard process analysis on visual recordings of business activities.
  • Few-shot labeling allows new event types to be added with limited examples after initial setup.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Live video feeds could feed this pipeline for ongoing process monitoring rather than post-hoc analysis.
  • Combining the video-derived events with audio or sensor data might produce richer multi-modal logs.
  • The segmentation step could be tested on videos with overlapping activities to check separation accuracy.

Load-bearing premise

Pre-trained image embeddings combined with frame-wise similarity matrices will reliably detect and separate meaningful process events across different video domains and lighting conditions without extensive per-video tuning.

What would settle it

A collection of videos from a new domain or under different lighting where the generated event logs show mismatched event sequences or timings compared to manual annotations of the same footage.

Figures

Figures reproduced from arXiv: 2604.22476 by Dustin Heller, Jonas Seng, Kristian Kersting, Marco Pegoraro, Wil M.P. van der Aalst.

Figure 1
Figure 1. Figure 1: SnapLog extracts logs from video data. Our method, SnapLog, constructs an event log from video data which can then be used downstream by standard process mining algorithms. checkpoints or baggage handling systems. Rather than being recorded directly as event logs, these heterogeneous data sources cannot directly describe operations in processes such as check-in, security screening, boarding, or baggage tra… view at source ↗
Figure 2
Figure 2. Figure 2: Video Segmentation. From left to right: After the extraction of video features using a ViT, the feature vectors are used to compute a similarity matrix. By applying K-Means on the similarity matrix, atomic events are formed. These are merged into larger action segments using a greedy merging algorithm. few new examples [5,22]. In our pipeline, we adopt a few-shot video classifica￾tion technique inspired by… view at source ↗
Figure 3
Figure 3. Figure 3: Finetuning. Given a few labeled examples for novel classes, we first perform data augmentation by applying transformations (gray-scale, rotations, etc.) to increase the number of given labeled examples. Then, a new head is added to the R(2+1)D encoder and learned via gradient descent. 3.2 Few-Shot Video Classification Event Classification To classify the identified temporal events, we employ a flexible few… view at source ↗
Figure 4
Figure 4. Figure 4: Silhouette scores results. The average silhouette scores of a single video from the TUM dataset for k = 3, 5, and 7 clusters (average represented by the red line). The thickness of the cluster bars indicates the cluster size and the length of the cluster indicates a larger separation to other clusters. k = 7, although having a lower average silhouette score, provides the most balanced clustering for our purposes view at source ↗
Figure 5
Figure 5. Figure 5: Our pipeline produces semantically meaningful video segmentations. view at source ↗
Figure 6
Figure 6. Figure 6: A resulting process model. An example of directly-follows graph extracted from our validation in the non-augmented experiment on the TUM dataset. Red com￾ponents show faults. In summary, despite some inaccuracies, a clear flow of the process of baking brownies can be identified through the model, which represents the overarching process throughout the videos relatively reliably. 5 Conclusion This paper pre… view at source ↗
read the original abstract

Disciplines such as business process management and process mining aid organizations by discovering insights about processes on the basis of recorded event data. However, an obstacle to process analysis is data multi-modality: for instance, data in video form are not directly interpretable as events. In this work, we present SnapLog, an approach to extract event data from videos by converting frames to feature vectors using image embeddings and performing temporal segmentation through frame-wise similarity matrices. A generalized few-shot classification is then used to assign labels to the video segments, yielding labeled, timestamped sub-sequences of frames that are interpretable as events. Conventional process mining techniques can be used to analyze the resulting data. We show that our approach produces logs that accurately reflect the process in the videos.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The manuscript presents SnapLog, a pipeline to extract event logs from video streams for process mining. Frames are converted to feature vectors via pre-trained image embeddings; temporal segmentation is performed using frame-wise similarity matrices; generalized few-shot classification then labels the resulting segments to produce timestamped, interpretable events. The authors claim that the resulting logs accurately reflect the underlying processes depicted in the videos.

Significance. If empirically validated across domains, the work could meaningfully advance process mining by automating the conversion of ubiquitous video data into usable event logs, addressing a key data-acquisition bottleneck in business process management without requiring manual annotation.

major comments (2)
  1. Abstract: the central claim that the approach 'produces logs that accurately reflect the process in the videos' is unsupported by any quantitative metrics, baselines, error rates, or evaluation details. This absence makes the soundness of the core assertion impossible to assess from the manuscript.
  2. Method (segmentation and classification steps): the pipeline relies on the untested assumption that generic pre-trained embeddings plus cosine-similarity matrices will reliably detect event boundaries and that few-shot labeling will assign correct process labels across varying lighting, viewpoints, and domains without per-video tuning or noise handling; no robustness experiments or failure-case analysis are described to support this load-bearing generalization.
minor comments (1)
  1. Abstract: a concise statement of the evaluation datasets or video domains used would help readers immediately gauge the scope of the accuracy claim.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on our manuscript. The comments highlight important areas where the presentation and validation of our claims can be strengthened. We address each major comment below and describe the revisions we will make to the next version of the paper.

read point-by-point responses

  1. Referee: Abstract: the central claim that the approach 'produces logs that accurately reflect the process in the videos' is unsupported by any quantitative metrics, baselines, error rates, or evaluation details. This absence makes the soundness of the core assertion impossible to assess from the manuscript.

    Authors: We agree that the abstract's phrasing would benefit from more precise language and supporting evidence. The current manuscript presents the pipeline and provides illustrative examples of the resulting event logs, but it does not include formal quantitative evaluation. We will revise the abstract to state that the approach yields interpretable event logs and will add a dedicated Experiments section that reports quantitative metrics (e.g., boundary detection F1, label accuracy), comparisons against baselines, and error rates on held-out video sequences. revision: yes

  2. Referee: Method (segmentation and classification steps): the pipeline relies on the untested assumption that generic pre-trained embeddings plus cosine-similarity matrices will reliably detect event boundaries and that few-shot labeling will assign correct process labels across varying lighting, viewpoints, and domains without per-video tuning or noise handling; no robustness experiments or failure-case analysis are described to support this load-bearing generalization.

    Authors: The method section motivates the use of pre-trained embeddings and similarity matrices by referencing their established performance in related vision tasks, and the generalized few-shot component is intended to reduce the need for per-video tuning. Nevertheless, we acknowledge that explicit robustness validation is missing. We will add a new subsection with experiments that systematically vary lighting conditions, camera viewpoints, and process domains, report performance degradation, and include a failure-case analysis (e.g., cases where similarity thresholds produce over- or under-segmentation). revision: yes

Circularity Check

0 steps flagged

No circularity: forward pipeline with empirical claim

full rationale

The paper presents SnapLog as a sequence of independent processing steps (pre-trained embeddings to feature vectors, similarity-matrix segmentation, generalized few-shot labeling) that produce event logs, followed by an empirical claim that the logs accurately reflect the video processes. No equations, fitted parameters, or self-citations are shown to reduce the central result to its own inputs by construction. The derivation is a standard modular pipeline whose validity rests on external validation rather than self-definition or renaming.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The approach rests on standard computer-vision assumptions about embeddings and similarity without introducing new entities or many free parameters; evaluation details are absent so the ledger remains minimal.

axioms (2)
  • domain assumption Pre-trained image embeddings produce feature vectors whose similarity reflects semantically meaningful process steps.
    Invoked by the use of embeddings for frame-wise feature extraction and subsequent similarity matrices.
  • domain assumption Temporal segmentation via frame similarity matrices will align with actual event boundaries in workflow videos.
    Central to the segmentation step described in the abstract.

pith-pipeline@v0.9.0 · 5439 in / 1166 out tokens · 45996 ms · 2026-05-08T12:41:30.170353+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 1 internal anchor

  1. [1]

    In: Problems@BPM

    Cohen, I., Gal, A.: Uncertain process data with probabilistic knowledge: Problem characterization and challenges. In: Problems@BPM. CEUR Workshop Proceed- ings, vol. 2938, pp. 51–56. CEUR-WS.org (2021)

    work page 2021

  2. [2]

    International Journal of Computer Vision pp

    Damen, D., Doughty, H., Farinella, G.M., Furnari, A., Kazakos, E., Ma, J., Molti- santi, D., Munro, J., Perrett, T., Price, W., et al.: Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens-100. International Journal of Computer Vision pp. 1–23 (2022)

    work page 2022

  3. [3]

    IEEE Transactions on Pattern Analysis and Machine Intelligence 46(2), 1011–1030 (2023)

    Ding, G., Sener, F., Yao, A.: Temporal action segmentation: An analysis of mod- ern techniques. IEEE Transactions on Pattern Analysis and Machine Intelligence 46(2), 1011–1030 (2023)

    work page 2023

  4. [4]

    An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020) 16 M. Pegoraro et al

    work page internal anchor Pith review arXiv 2010

  5. [5]

    Ferdaus, M.M., Niles, K.N., Tom, J., Abdelguerfi, M., Ioup, E.: Few-shot learning in video and 3d object detection: A survey (2025)

    work page 2025

  6. [6]

    In: ICPM

    Gal, A.: Everything there is to know about stochastically known logs. In: ICPM. pp. xvii–xxiii. IEEE (2023)

    work page 2023

  7. [7]

    In: 2024 26th Inter- national Conference on Business Informatics (CBI)

    Gavric, A., Bork, D., Proper, H.: Multimodal process mining. In: 2024 26th Inter- national Conference on Business Informatics (CBI). pp. 99–108. IEEE (2024)

    work page 2024

  8. [8]

    In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition

    Karpathy,A.,Toderici,G.,Shetty,S.,Leung,T.,Sukthankar,R.,Fei-Fei,L.:Large- scale video classification with convolutional neural networks. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 1725–1732 (2014)

    work page 2014

  9. [9]

    In: Modellierung

    Koschmider, A., Aleknonyte-Resch, M., Fonger, F., Imenkamp, C., Lepsien, A., Apaydin, K., Janssen, D., Langhammer, D., Ziolkowski, T., Zisgen, Y.: Process mining for unstructured data: Challenges and research directions. In: Modellierung. LNI, vol. P-348, pp. 119–136. Gesellschaft für Informatik e.V. (2024)

    work page 2024

  10. [10]

    In: BPM (Forum)

    Lepsien, A., Koschmider, A., Kratsch, W.: Analytics pipeline for process mining on video data. In: BPM (Forum). Lecture Notes in Business Information Processing, vol. 490, pp. 196–213. Springer (2023)

    work page 2023

  11. [11]

    Madan, N., Moegelmose, A., Modi, R., Rawat, Y.S., Moeslund, T.B.: Foundation models for video understanding: A survey (2024)

    work page 2024

  12. [12]

    Agriculture13(8) (2023)

    Melfsen, A., Lepsien, A., Bosselmann, J., Koschmider, A., Hartung, E.: Describ- ing behavior sequences of fattening pigs using process mining on video data and automated pig behavior recognition. Agriculture13(8) (2023)

    work page 2023

  13. [13]

    In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision

    Nam, J., Ahn, D., Kang, D., Ha, S.J., Choi, J.: Zero-shot natural language video localization. In: Proceedings of the IEEE/CVF International Conference on Com- puter Vision. pp. 1470–1479 (2021)

    work page 2021

  14. [14]

    In: CAiSE (Doctoral Con- sortium)

    Pegoraro, M.: Probabilistic and non-deterministic event data in process mining: Embedding uncertainty in process analysis techniques. In: CAiSE (Doctoral Con- sortium). CEUR Workshop Proceedings, vol. 3139, pp. 37–46. CEUR-WS.org (2022)

    work page 2022

  15. [15]

    Information Systems102, 101810 (2021)

    Pegoraro, M., Uysal, M.S., van der Aalst, W.M.P.: Conformance checking over uncertain event data. Information Systems102, 101810 (2021)

    work page 2021

  16. [16]

    ACM Computing Surveys (2023)

    Schiappa, M.C., Rawat, Y.S., Shah, M.: Self-supervised learning for videos: A sur- vey. ACM Computing Surveys (2023)

    work page 2023

  17. [17]

    In: Proceedings of the 1st International Conference on Process Mining, Aachen, Germany

    Seeliger, A., Maximilian, R., Nolle, T., Mühlhäuser, M., Andrea, M., Artem, P., van Sebastiaan, Z.: Process explorer: Interactive visual exploration of event logs with analysis guidance. In: Proceedings of the 1st International Conference on Process Mining, Aachen, Germany. pp. 24–26 (2019)

    work page 2019

  18. [18]

    In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops

    Tenorth, M., Bandouch, J., Beetz, M.: The TUM kitchen data set of everyday manipulation activities for motion tracking and action recognition. In: 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops. pp. 1089–1096. IEEE (2009)

    work page 2009

  19. [19]

    In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition

    Tran, D., Wang, H., Torresani, L., Ray, J., LeCun, Y., Paluri, M.: A closer look at spatiotemporal convolutions for action recognition. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 6450–6459 (2018)

    work page 2018

  20. [20]

    IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    Wang, B., Zhao, Y., Yang, L., Long, T., Li, X.: Temporal action localization in the deep learning era: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    work page 2024

  21. [21]

    arXiv preprint arXiv:2308.01618 (2023)

    Wang, P., Zeng, F., Qian, Y.: A survey on deep learning-based spatio-temporal action detection. arXiv preprint arXiv:2308.01618 (2023)

    work page arXiv 2023

  22. [22]

    Wanyan, Y., Yang, X., Dong, W., Xu, C.: A comprehensive review of few-shot action recognition (2025) Automated and Efficient Event Discovery from Video Streams 17

    work page 2025

  23. [23]

    IEEE Transactions on Pattern Analysis and Machine Intelligence44(12), 8949– 8961 (2021)

    Xian, Y., Korbar, B., Douze, M., Torresani, L., Schiele, B., Akata, Z.: Gener- alized few-shot video classification with video retrieval and feature generation. IEEE Transactions on Pattern Analysis and Machine Intelligence44(12), 8949– 8961 (2021)

    work page 2021