Pith. sign in

REVIEW 4 major objections 5 minor 16 references

Mamba-OTR: a Mamba-based Solution for Online Take and Release Detection from Untrimmed Egocentric Video

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Mamba-OTR claims that a Mamba backbone trained on short clips detects take/release endpoints in egocentric video with 45.48 mp-mAP sliding and 43.35 in streaming, far above vanilla transformer (20.32) and vanilla Mamba (25.16).

desk verdict Fixed-window regularization is a genuinely useful trick, but the headline numbers contradict each other (51.76 vs 45.48) and the timing is uninterpretable; worth a serious referee after the authors reconcile the results. read the letter →

arxiv 2507.16342 v1 pith:UW2FDCHP submitted 2025-07-22 cs.CV

Alessandro Sebastiano Catinello , Giovanni Maria Farinella , Antonino Furnari This is my paper
classification cs.CV
keywords onlineactiondetectiontake/releaseegocentricvideoMambaselectivestatespacemodelsfocallosstemporalregularizationstreaminginference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that online detection of the exact moment an object is taken or released in egocentric video can be done accurately and efficiently by a Mamba-based model trained on short clips and run recurrently over full videos. It argues that the extreme class imbalance—one positive frame per action among thousands of background frames—is best handled by focal loss plus a fixed-window regularizer that suppresses duplicate predictions around ground-truth endpoints. On the EPIC-KITCHENS-100 take/release subset, the resulting Mamba-OTR reaches 45.48 mean point-level mAP with sliding-window inference and 43.35 in true streaming mode, versus 20.32 for a vanilla transformer and 25.16 for vanilla Mamba. If correct, the work shows that recurrence-based architectures decouple training length from inference length, so cheap training on 5-second snippets does not cost accuracy on full-length video, an important property for real-time wearable assistance.

What carries the argument

The load-bearing machinery is the Mamba selective state-space layer, a recurrent sequence model that processes tokens in linear time and maintains a hidden state at inference, so a model trained on short windows can be unrolled over a full video without re-setting the state or re-processing frames. Around it, the paper builds a training pipeline: focal loss to counter the one-positive-frame-per-action imbalance, and a fixed-window regularizer that sums predicted probabilities in a 4-frame window centered on each ground-truth take/release end frame, explicitly teaching the model to emit one confident spike per event to match the point-level mAP metric.

What would settle it

Run the released Mamba-OTR in streaming mode on videos substantially longer than those in EPIC-KITCHENS-100, or at a different frame rate, and measure mp-mAP: if accuracy degrades with sequence length, or if streaming frame-level predictions diverge from chunked predictions on identical frames, the training-inference decoupling claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a Mamba model trained on fixed 20-frame (5-second) chunks can be applied at test time to full-length videos in a fully streaming, frame-by-frame manner, retaining most of its accuracy (43.35 vs 45.48 mp-mAP) while a transformer collapses to 0.04 when moved from sliding-window to streaming evaluation. The paper attributes this to Mamba's recurrent hidden state, which carries temporal context across arbitrarily long sequences without the positional-embedding mismatch that breaks transformers. Together with focal loss and a fixed-window regularization that penalizes extra predictions near each ground-truth endpoint, this training-inference decoupling yields the reported gains: 41.01 with focal loss alone, 42.98 with 20-frame inputs, and 45.48 after regularization.

Load-bearing premise

The claim rests on the assumption that a Mamba model trained on fixed 20-frame clips can be run in streaming mode over full-length videos, carrying its hidden state across the whole sequence without harmful distribution shift, a reliance supported only by the single sliding-vs-streaming comparison in Table 7.

Editorial extensions

If this is right

  • Mamba-OTR can be deployed in a true streaming setting, processing each frame once with roughly 8 nanoseconds of overhead per frame beyond feature extraction and no frame buffer, making it suitable for real-time wearable assistance.
  • Training on 5-second clips (20 frames) is sufficient; increasing the training input to 10 seconds does not help, consistent with the observation that take/release actions last about 2 seconds on average.
  • Focal loss is necessary for all architectures tested, improving Mamba from 25.16 to 41.01 mp-mAP, and the fixed-window regularizer adds a further gain to 45.48.
  • Transformer-based online detectors cannot simply be switched from windowed to streaming inference, as their accuracy collapses from 38.96 to 0.04, so streaming deployment requires recurrence.
  • The fixed-window regularizer also improves transformer baselines, indicating that metric-aligned training is beneficial beyond the Mamba architecture.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the training-inference length decoupling likely extends to other sparse, point-labeled temporal tasks such as action start detection, anomaly events, or gesture endpoints, where positive frames are rare and temporal precision matters.
  • The paper does not explore hidden-state drift on very long recordings; a stress test on hour-long streams or after many actions could reveal whether streaming accuracy degrades below the reported 43.35 over time.
  • The fixed-window regularizer could be adapted to soft labels, such as Gaussian windows around endpoints, or to other point-level metrics, a direct extension of the loss design that the paper leaves implicit.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes Mamba-OTR, a Mamba-based model for online take/release endpoint detection in untrimmed egocentric video. The method combines the Mamba state-space architecture with focal loss and a fixed-window regularization term that penalizes multiple predictions around ground-truth action-ending frames. Experiments are conducted on a curated subset of EPIC-KITCHENS-100, comparing TeSTra, a Transformer baseline, and vanilla Mamba. The authors report an mp-mAP of 45.48 in sliding-window mode and 43.35 in streaming mode, versus 20.32 for a vanilla Transformer and 25.16 for a vanilla Mamba, and claim that Mamba's recurrence allows training on short clips while performing efficient streaming inference over full-length videos. Section 1 additionally reports a conflicting headline number of 51.76 mp-mAP.

Significance. If the reported numbers are reproducible, Mamba-OTR would be a strong and lightweight baseline for online take/release detection. The fixed-window regularizer is simple and directly motivated by the p-mAP greedy-matching rule, which is a point in its favor. The idea of decoupling training clip length from inference sequence length is practically relevant for egocentric online systems. However, the manuscript currently presents inconsistent headline numbers, no error bars, and no released code or evaluation protocol, so the significance is not yet fully assessable from the paper as written.

major comments (4)
  1. [Section 1 vs. Abstract/Tables 4, 5, 7] The central quantitative claim is internally inconsistent. Section 1 states that the optimized Mamba-OTR achieves an mp-mAP of 51.76 and a mean inference time of 0.14s, versus 20.32 and 0.28s for a standard Transformer. The abstract and Tables 4, 5, and 7 report Mamba-OTR as 45.48 in sliding-window mode and 43.35 in streaming mode. The 6.28-point gap between 51.76 and 45.48 is larger than the 2.06-point margin used to justify the fixed-window regularizer over the sliding-window variant, so the main result is undefined as printed. Please correct the inconsistency and specify exactly which configuration (input length, regularization, inference mode) each number refers to.
  2. [Table 7] The efficiency claim is uninterpretable. The table reports a per-frame inference time of 8ns for both models and a per-video time of 0.14s, which would require approximately 17.5 million video frames. Section 1 also credits the Transformer with 0.28s per video, while Table 7 gives both models 0.14s. Please report the measurement protocol (hardware, batch size, sequence length, number of videos averaged) and give times in consistent units, or remove the per-frame row.
  3. [Section 5.4] The main qualitative claim that Mamba-trained models generalize from 20-frame chunks to full-video streaming rests on a single comparison in Table 7. There is no analysis of hidden-state drift, no sensitivity to sequence length or frame rate, and no error bars. Given that the Transformer's streaming mp-mAP collapses to 0.04, a number that is not explained, the streaming claim is not yet established. Please add experiments varying video length and frame rate, and report variance over runs.
  4. [Sections 5.1-5.3] Hyperparameters are selected on the reported test metric without a validation split. The number of Mamba layers, training input length, regularization type, and window size are all chosen based on the final mp-mAP numbers in Tables 2-5. With differences as small as 0.35-1.35 points and no error bars, the reported improvements may be within run-to-run noise. Please provide a validation-based selection protocol or confidence intervals.
minor comments (5)
  1. [Abstract and Introduction] There are several typos: 'These finding' should be 'These findings', 'Tranformers' should be 'Transformers', and 'the20.32' should be 'the 20.32'.
  2. [Section 5.3] 'Tab.5ablatestheeffectiveness' should read 'Tab. 5 ablates the effectiveness'; also, 'MAMBA-OTR' capitalization is inconsistent.
  3. [Section 4.2, Eq. (1)] The summation notation in Equation (1) is garbled and should be typeset correctly.
  4. [Section 5.1] The text says 'All models use 1 second (4 frames) for the short memory,' but Transformer and Mamba are described as having no long memory; please clarify how the short memory is applied for these models.
  5. [References] The paper relies on [3] for the dataset, the ODAE formulation, and baseline numbers; since [3] is a self-citation, please clarify the relationship and ensure no double-counting of results.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the fixed-window regularizer is an explicit training objective tied to the metric, and self-citations to [3] are dataset/task dependencies, not reductions of the central result.

full rationale

Mamba-OTR's central claims are empirical measurements (mp-mAP values in Tables 1-7) obtained by training and testing the proposed architecture on EPIC-KITCHENS-100. The only step that could look construction-level is the Fixed Window regularization (Eq. 4), which sums predicted probabilities in windows centered on ground-truth ending frames. This is not presented as a derived prediction; Section 5.3 states it is explicitly designed to encourage a single high-confidence prediction per ground truth because the p-mAP metric "penalize[s] all other predictions made in its vicinity." A loss term deliberately built to match the evaluation metric is transparent training design, not circularity: the reported gain (42.98 to 45.48, Table 4) is an empirical outcome, not an identity. Self-citations to [3] provide the ODAE formulation, the curated dataset, and the baseline configurations reproduced in Table 1; these are inputs/dependencies, not a derivation chain that reduces the new result to those citations. The Mamba-OTR scores and the streaming result in Table 7 are new measurements, and no equation equates them to the inputs. The manuscript does contain a numeric inconsistency (Section 1 reports 51.76 mp-mAP and 0.14s while the abstract and Tables 4/7 report 45.48/43.35; Table 7's 8ns per-frame time is incompatible with 0.14s per video), but inconsistency is a correctness/reproducibility issue, not circularity. Hence score 0.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central empirical results rest on a handful of hyperparameters tuned by ablation on the reported test metric, on the precomputed feature representation from [12], and on the ODAE formulation inherited from the authors' own prior work [3]. No new entities are introduced. The most fragile dependency is the train-short/infer-long generalization of Mamba's recurrence, which is assumed rather than analyzed.

free parameters (5)
  • Focal loss gamma and alpha = not reported
    Focal loss is central to the reported improvements (Section 5.1), but the gamma and alpha values are not given in the paper.
  • Regularization weight lambda = 0.01
    Set by hand in Section 5.3 with no sensitivity analysis reported.
  • Fixed window size = 4 frames (1 second)
    Chosen by ablation on the reported mp-mAP (Table 5), optimal among 4, 12, and 20 frames.
  • Number of Mamba layers = 3
    Chosen by ablation on the reported mp-mAP (Table 2); 3 layers performed best.
  • Training input length = 20 frames (5 seconds)
    Chosen by ablation on the reported mp-mAP (Table 3), best among 4, 8, 12, 20, and 40 frames.
assumptions (4)
  • domain assumption The ODAE formulation (predicting the ending frame of an action) is the appropriate target for OTR
    Adopted from self-cited [3] without independent comparison to start-frame or contact-frame formulations; the entire evaluation rests on this choice.
  • domain assumption High-level features from [12] sampled at 4 fps are a sufficient representation for detecting take/release endpoints
    All models are trained and evaluated on these precomputed features, so results are conditional on their quality.
  • domain assumption A Mamba model trained on fixed-length chunks generalizes to full-length streaming sequences
    Section 5.4 assumes hidden-state recurrence carries over to arbitrary sequence lengths; no analysis of drift or length sensitivity is provided.
  • domain assumption Per-action duration statistics (about 8 frames or 2 seconds on average) from [3] justify the chosen window sizes
    Used to motivate the 20-frame training window and the 4-frame fixed window in Sections 5.2 and 5.3.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mamba-OTR: a Mamba-based Solution for Online Take and Release Detection from Untrimmed Egocentric Video." pith.science (2026). https://pith.science/paper/UW2FDCHP

@misc{pith2026250716342,
  author       = {Pith},
  title        = {Pith review of: Mamba-OTR: a Mamba-based Solution for Online Take and Release Detection from Untrimmed Egocentric Video},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UW2FDCHP}},
  note         = {Machine review of arXiv:2507.16342}
}
read the original abstract

This work tackles the problem of Online detection of Take and Release (OTR) of an object in untrimmed egocentric videos. This task is challenging due to severe label imbalance, with temporally sparse positive annotations, and the need for precise temporal predictions. Furthermore, methods need to be computationally efficient in order to be deployed in real-world online settings. To address these challenges, we propose Mamba-OTR, a model based on the Mamba architecture. Mamba-OTR is designed to exploit temporal recurrence during inference while being trained on short video clips. To address label imbalance, our training pipeline incorporates the focal loss and a novel regularization scheme that aligns model predictions with the evaluation metric. Extensive experiments on EPIC-KITCHENS-100, the comparisons with transformer-based approach, and the evaluation of different training and test schemes demonstrate the superiority of Mamba-OTR in both accuracy and efficiency. These finding are particularly evident when evaluating full-length videos or high frame-rate sequences, even when trained on short video snippets for computational convenience. The proposed Mamba-OTR achieves a noteworthy mp-mAP of 45.48 when operating in a sliding-window fashion, and 43.35 in streaming mode, versus the 20.32 of a vanilla transformer and 25.16 of a vanilla Mamba, thus providing a strong baseline for OTR. We will publicly release the source code of Mamba-OTR to support future research.

Figures

Figures reproduced from arXiv: 2507.16342 by the authors.

Figure 1
Figure 1. Algorithms are tasked to process the video online and output a single prediction corresponding to the last frame of the take/release action (blue frame), while avoiding predictions for background (red frame) or any other frame (green ones). recognition algorithms must operate on streaming video data in an online fashion while maintaining temporal coherence by emitting a single, unambiguous predic￾tion per action ins… view at source ↗
Figure 2
Figure 2. Models architecture overview. a) TeSTra uses an encoder for long-term memory and a decoder to combine long- and short-term information. b) Transformer processes only short-term memory using standard attention blocks. c) MAMBA-OTR employs one or more Mamba layers for efficient temporal modeling. TeSTra We consider the TeSTra architecture [16] as a state-of-the-art Transformer￾based model optimized for online video pr… view at source ↗
Figure 3
Figure 3. (a) Sliding Window: a window slides on all frames, encouraging sparse predic￾tions. (b) Fixed Window: we place windows only around ground truth actions encour￾aging sparse predictions only in those parts of the video. 5 Experiments In this section we consider the dataset introduced in [3], which is a curated subset of EPIC-KITCHENS-100, containing only two action classes: take (e.g., “get”, “fetch”) and release (e.g… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison between different regularization techniques. regularization with a window size of 4 frames. We note that both architectures introduce minimal computational overhead beyond the backbone, both requiring only around 0.14 seconds per video and 8 nano…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

16 extracted references · 14 canonical work pages

  1. [1]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)

    An, J., Kang, H., Han, S.H., Yang, M.H., Kim, S.J.: Miniroad: Minimal rnn frame- work for online action detection. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (2023)

    work page 2023

  2. [2]

    In: Advances in Neural Information Pro- cessing Systems (NeurIPS) (2019)

    Cao, K., Wei, C., Gaidon, A., Arechiga, N., Ma, T.: Learning imbalanced datasets with label-distribution-aware margin loss. In: Advances in Neural Information Pro- cessing Systems (NeurIPS) (2019)

    work page 2019

  3. [3]

    In: International Conference on Computer Vision Theory and Applications (2025)

    Catinello, A.S., Farinella, G.M., Furnari, A.: Online detection of end of take and release actions from egocentric videos. In: International Conference on Computer Vision Theory and Applications (2025)

    work page 2025

  4. [4]

    arXiv preprint arXiv:2403.09626 (2024)

    Chen, G., Huang, Y., Xu, J., Pei, B., Chen, Z., Li, Z., Wang, J., Li, K., Lu, T., Wang, L.: Video mamba suite: State space model as a versatile alternative for video understanding. arXiv preprint arXiv:2403.09626 (2024)

    arXiv 2024

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    Cui, Y., Jia, M., Lin, T.Y., Song, Y., Belongie, S.: Class-balanced loss based on effective number of samples. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2019)

    work page 2019

  6. [6]

    In: ICCV (2019)

    Gao, M., Xu, M., Davis, L.S., Socher, R., Xiong, C.: Startnet: Online detection of action start in untrimmed videos. In: ICCV (2019)

    work page 2019

  7. [7]

    In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (2021)

    Gao, M., Zhou, Y., Xu, R., Socher, R., Xiong, C.: Woad: Weakly supervised online action detection in untrimmed videos. In: Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (2021)

    work page 2021

  8. [8]

    arXiv preprint arXiv:2312.00752 (2023)

    Gu, A., Dao, T.: Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 (2023)

    arXiv 2023

Show all 16 references
  1. [9]

    In: IEEE International Conference on Computer Vision (2017)

    Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: IEEE International Conference on Computer Vision (2017)

    2017

  2. [10]

    In- ternational Journal of Computer Vision (2024)

    Plizzari, C., Goletto, G., Furnari, A., Bansal, S., Ragusa, F., Farinella, G.M., Damen, D., Tommasi, T.: An outlook into the future of egocentric vision. In- ternational Journal of Computer Vision (2024)

    2024

  3. [11]

    In: International Conference on Image Analysis and Processing (2023)

    Scavo, R., Ragusa, F., Farinella, G.M., Furnari, A.: Quasi-online detection of take and release actions from egocentric videos. In: International Conference on Image Analysis and Processing (2023)

    2023

  4. [12]

    In: ECCV (2018)

    Shou, Z., Pan, J., Chan, J., Miyazawa, K., Mansour, H., Vetro, A., Giro-i Nieto, X., Chang, S.F.: Online detection of action start in untrimmed, streaming videos. In: ECCV (2018)

    2018

  5. [13]

    Wang, X., Zhang, S., Qing, Z., Shao, Y., Zuo, Z., Gao, C., Sang, N.: Oadtr: Online actiondetectionwithtransformers.In:ProceedingsoftheIEEE/CVFInternational Conference on Computer Vision (2021)

    2021

  6. [14]

    In: ICCV (2019)

    Xu, M., Gao, M., Chen, Y.T., Davis, L.S., Crandall, D.J.: Temporal recurrent networks for online action detection. In: ICCV (2019)

    2019

  7. [15]

    NeurIPS (2021)

    Xu, M., Xiong, Y., Chen, H., Li, X., Xia, W., Tu, Z., Soatto, S.: Long short-term transformer for online action detection. NeurIPS (2021)

    2021

  8. [16]

    In: European Conference on Computer Vision (2022)

    Zhao, Y., Krähenbühl, P.: Real-time online video detection with temporal smooth- ing transformers. In: European Conference on Computer Vision (2022)

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.