Pith. sign in

REVIEW 3 major objections 5 minor 62 references

HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HiERO shows that hierarchical activity threads, learned from narrations alone, make egocentric video features strong enough to beat supervised procedure learning in zero shot.

desk verdict A competent hierarchical-graph method with genuinely useful ablations, undermined by the abstract's unsupported claim of beating fully-supervised methods. read the letter →

arxiv 2505.12911 v1 pith:IO6FMK2Y submitted 2025-05-19 cs.CV

Simone Alberto Peirone , Francesca Pistilli , Giuseppe Averta This is my paper · ORCID
classification cs.CV
keywords egocentricvideounderstandinghierarchicalactivitystructureprocedurelearningweakly-supervisedvideo-textalignmentspectralclusteringzero-shotsteplocalizationgraphneuralnetworksfunctionalthreads
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HiERO argues that human activities in unscripted egocentric video carry a hidden hierarchy: individual actions group into functional threads (e.g., prepping vegetables, cooking pasta), and those threads group into broader routines. The paper claims this hierarchy can be learned without step-level supervision, using only the alignment between short video clips and their narrated descriptions, and that enriching segment features with these multi-scale groupings improves downstream reasoning. On procedure learning, the enriched features allow steps to be recovered by clustering alone: HiERO reports state-of-the-art results in zero shot, outperforming fully-supervised methods by +12.5% F1 on EgoProceL. The same features also improve video-text alignment benchmarks (EgoMCQ, EgoNLQ) with minimal extra training. A sympathetic reader would care because it suggests that the bottleneck for understanding long, unscripted human activity is not more supervision but the right inductive bias about how actions organize.

What carries the argument

The central mechanism is a spectral-clustering-based 'Cut & Match' module inside a Graph U-Net-style encoder-decoder. The video is a temporal graph of segment nodes; the Temporal Encoder aggregates information within local temporal neighborhoods at progressively coarser scales, while the Function-Aware Decoder, at each stage, partitions the current node set into K clusters by spectral clustering on a cosine-similarity graph, runs temporal reasoning inside each cluster separately, then reassembles the graph. This is what makes actions that are temporally distant but functionally related (e.g., 'cut onion' and 'cut carrot') become neighbors in feature space. Training combines a windowed video-narration contrastive loss, which makes co-occurring actions similar, with a functional-thread contrastive loss, which sharpens the clusters discovered by Cut & Match.

What would settle it

Re-run the EgoProceL zero-shot experiment with K set to each video's ground-truth number of key-steps (available in Table 6 of the paper) and compare F1 to the fixed-K=7 result; alternatively, sweep K for each video and check whether the F1 peak tracks the true step count. If performance depends on providing the correct K, the claim that steps emerge without task-specific supervision is weakened.

Watch

Extended reading notes

Core claim

On its own terms, the paper's claim is that hierarchical functional structure is not an annotation to be provided but a pattern that can emerge from data. HiERO models a long video as a graph whose nodes are fixed-length segments, trains an encoder-decoder that propagates information through temporally local neighborhoods while repeatedly cutting the graph into functionally similar clusters and matching nodes across those clusters, and supervises the whole process with a contrastive video-narration alignment loss applied at the decoder output and a contrastive loss that pulls same-cluster segments together. After training, procedure steps are detected by running spectral clustering on the learned segment features, with no task-specific fine-tuning. The paper reports that this zero-shot pipeline outperforms fully-supervised procedure learning baselines on EgoProceL (+12.5% F1), and that the same features improve EgoMCQ intra accuracy and EgoNLQ recall over strong video-text pretraining backbones.

Load-bearing premise

The load-bearing premise is that procedure steps show up as clean clusters in the learned feature space, yet the zero-shot evaluation fixes the cluster count at k=7 on EgoProceL while the benchmark's videos vary from 4 to 17 key-steps; if the true step count differs from K, spectral clustering cannot recover the correct segmentation and no mechanism in HiERO estimates K from the video content.

Editorial extensions

If this is right

  • Procedure learning becomes a zero-shot clustering operation: given enough narrated egocentric video, key-steps can be recovered from segment features without a single step-level annotation.
  • The same learned features transfer across tasks, so a single pretraining pass can support multiple-choice video-text retrieval, natural language temporal grounding, step grounding, and step localization.
  • Because the method does not need multiple scripted demonstrations of the same procedure, it can be applied to unscripted daily-life video where the same task is never performed twice in the same way.
  • The improvements hold across several base feature extractors, suggesting that the hierarchical enrichment is complementary to the choice of backbone.
  • If the zero-shot results are correct, weakly-supervised hierarchy discovery should become a standard component of egocentric video representation learning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: the reported zero-shot procedure-learning results fix the cluster count at k=7 on EgoProceL even though benchmark videos contain between 4 and 17 key-steps, so an adaptive estimator of K is a natural extension and the strength of the 'zero-shot' claim depends on whether performance survives when K is not tuned or known.
  • Not tested in the paper: the step-localization failures shown are mostly granularity mismatches with the annotation taxonomy, so the method appears to discover clusters at its own natural abstraction level; aligning that level with an external step/substep taxonomy is a separate open problem.
  • Not tested in the paper: because step labels are assigned by cosine similarity between averaged cluster features and taxonomy text embeddings, using a language model to summarize a cluster's narrations before matching could sharpen zero-shot classification without changing the clustering.
  • Extrapolating beyond the paper: the hierarchy learned from narrations alone could plausibly transfer to non-egocentric video of human activity, since the training signal is co-occurrence of narrated actions rather than camera viewpoint; the paper does not test this.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. HiERO is a weakly supervised graph encoder-decoder that enriches egocentric video segment features with hierarchical 'functional threads'. A video is represented as a graph whose nodes are fixed-length segments; a temporal encoder aggregates local context, while a function-aware decoder uses spectral clustering (Cut&Match) to partition nodes into K groups and applies temporal message passing within each group. Training combines a video-narration contrastive loss Lvna and a cluster-contrastive loss Lft, using EgoClip narrations as supervision. The authors evaluate the resulting features on EgoMCQ and EgoNLQ with a learned grounding head, and in zero-shot on EgoProceL procedure learning and Ego4D Goal-Step step grounding and localization. The abstract claims state-of-the-art results across all benchmarks and a +12.5% F1 gain over fully-supervised methods on EgoProceL.

Significance. The underlying idea is interesting and timely: if activity hierarchies can emerge from unscripted video plus narrations, then procedure learning and step grounding could be done without step-level supervision. The paper has real strengths: the training signal is anchored to external narrations, the downstream evaluations are external benchmarks, the approach is relatively cheap (under 20 GPU hours), the code is released, and the ablations (Table 5) and clustering comparisons (Tables 10-11) are useful. The qualitative analysis of failure modes is also candid. However, the headline quantitative claims are substantially overstated relative to the evidence, and the procedure-learning numbers depend on a fixed cluster count that is not adapted to the data. The contribution is defensible after the claims are aligned with the experiments.

major comments (3)
  1. [Abstract; Sec. 4.1.2; Table 2] The abstract's central quantitative claim, that HiERO 'outperforms fully-supervised methods by a large margin (+12.5% F1 on EgoProceL) in zero shot', is not supported by the experiments. Table 2 compares HiERO with Random, CnC, GPL-2D, GPL, and OPEL, all of which are unsupervised or self-supervised procedure-learning methods; no fully-supervised baseline appears in the table or elsewhere in the paper. The +12.5% figure is exactly the difference between Ours (EgoVLP), 44.5, and OPEL, 32.0, so the comparison is to an unsupervised method, not a fully-supervised one. Please add explicit supervised baselines on EgoProceL or, if none are available under the same protocol, reword the abstract and Section 4.1.2 so that the claim is limited to the methods actually compared.
  2. [Sec. 4.1.2; Table 2; Table 6; Appendix B.2] All EgoProceL results use a fixed number of clusters K=7, while Table 6 reports that the true number of key-steps in the benchmark ranges from 4 (Greek Salad, Sandwich) to 17 (MECCANO). With K=7, spectral clustering cannot represent a 17-step procedure, so the reported F1 and IoU are measured under a constraint that is not derived from video content and may cap the achievable score on several datasets. Please report per-dataset results with K equal to the true key-step count (or with an estimated K) in addition to the fixed-K protocol, and discuss how the mismatch affects HiERO and the baselines. Similarly, Tables 3 and 4 do not state the K used for zero-shot step grounding and localization; Appendix B.2 only mentions scikit-learn spectral clustering. This makes the Goal-Step zero-shot results irreproducible and leaves open the possibility that an undisclosed K drives the outcome. Please report K and any tuning used for each task.
  3. [Abstract; Sec. 4.1.3; Tables 3 and 4] The abstract's statement that HiERO 'achieves state-of-the-art performance in all the benchmarks' is too broad. In Table 3, zero-shot HiERO (EgoVLP) reaches R@1 11.57 at mIoU@0.3, below the supervised EgoVLP baseline at 15.64; in Table 4, zero-shot HiERO reaches average mAP 8.7, below supervised EgoOnly at 13.6. The paper may intend 'state of the art among zero-shot methods', but that is not what the abstract and conclusions say. Please qualify every SOTA claim by setting (supervised vs zero-shot, fixed-K vs free-K) and by metric, since several metrics in Table 1 also do not improve over the backbone (e.g., EgoNLQ R@5 at mIoU@0.3 decreases from 22.38 to 21.12 with LAVILA features).
minor comments (5)
  1. [Sec. 3.2.1, Eq. (6)] The functional-threads loss uses cluster assignments from the model's own Cut&Match module on its own current features. Because the main evaluations and the Lvna anchor are external, I do not see circularity in the evaluation sense, but the paper should briefly discuss potential degenerate solutions (e.g., trivial cluster assignments) and how the combination with Lvna avoids them.
  2. [Sec. 3.1, Eqs. (3)-(4)] The notation for the edge weight w_ij and the sign term s_ij should be clarified: in Eq. (3), w_ij appears to be a scalar computed in Eq. (4), but the expression 'w_ij ⊙ x'_j' uses an elementwise symbol; please reconcile the dimensions or explain the broadcasting.
  3. [Appendix B.1] The approximation that propagates cluster labels from a temporally subsampled graph to the original graph by the temporally closest node is a strong assumption, since functional threads need not be temporally contiguous; please validate this approximation or state its expected effect on training.
  4. [Tables 10 and 11] The differences between spectral clustering and K-Means on EgoProceL are small (e.g., 44.5 vs 44.2 F1 for Ours EgoVLP), and no variance is reported; consider reporting multiple runs or significance in the clustering comparison.
  5. [Throughout] There are minor typographical issues: 'LoRa' should be 'LoRA', 'L AVILA' should be 'LAVILA', and the figure labels in Fig. 3 ('Sampling Interp.', 'Func. Thread Generation') do not exactly match the terms defined in the text ('Cut & Match module'); please align them.

Circularity Check

1 steps flagged · score 3.0 of 10

No load-bearing self-citation or fitted-input circularity; the only self-referential element is the functional-threads loss Lft, which reinforces the model's own spectral clusters and is a minor contributor to the externally validated results.

  1. self definitional [Section 3.2.1 (Functional threads loss, Eq. 6); Cut & Match module in Sec. 3.2; zero-shot inference in Sec. 3.3]
    "The functional threads loss Lft leverages the graph partition assignments from the Cut & Match modules in the decoder and pushes closer to each other samples that are assigned to the same cluster, while pushing away samples from other clusters."

    Eq. (6) defines Lft using the cluster assignments c_i (and c_j) produced by the Cut & Match module, which builds the partitions by spectral clustering on the same node embeddings h_v(v_i) that Eq. (6) then updates. The functional threads are therefore not independently observed: they are clusters of the model's own features, and the loss only makes those features more consistent with their own clustering. At inference (Sec. 3.3) the procedure steps are recovered by applying the same spectral clustering to the decoder outputs, so the 'emergence' of threads is partly a self-consistency loop.

full rationale

The central training signal of HiERO, Lvna, is anchored to external video narrations, and the main evaluation numbers come from external benchmarks (EgoProceL, Goal-Step, EgoMCQ, EgoNLQ). The zero-shot procedure-learning pipeline clusters decoder features and matches them to ground-truth key-steps with the Hungarian algorithm; the ground truth is not an input to the training loss. The self-citations to the authors' prior work (Refs. [37,38]) are not load-bearing: the TDGC update equations are stated in the paper, and no uniqueness theorem or prior result is used to forbid alternative architectures. The abstract's claim of outperforming fully-supervised methods by +12.5% F1 on EgoProceL is not supported by Table 2, which only lists unsupervised baselines, but that is a correctness/support issue, not circularity. The only genuinely self-referential step is Lft, which uses cluster assignments computed by the model's own Cut&Match on its own features and then pulls those same features closer within those clusters. This is a real self-training loop, but it is a minor contributor to the final numbers and does not make the main results equivalent to the inputs by construction. Hence a low-moderate circularity score of 3 is appropriate.

Assumptions & free parameters 8 free parameters · 5 assumptions · 1 invented entities

The central claims rest on several unproven domain assumptions about the link between co-occurrence, function, and human activity structure, plus a handful of hyperparameters (K, alpha, beta, kappa, tau, Delta) that are either tuned on the evaluation benchmarks or left unspecified. The functional threads concept is an invented construct operationalized entirely through the model's clustering, so it does not provide independent evidence for the hierarchy claim.

free parameters (8)
  • Number of clusters K (k=7) = 7 (EgoProceL); not reported (Goal-Step)
    Clustering is the core inference mechanism for zero-shot steps; EgoProceL uses a fixed k=7 for all tasks although true key-step counts vary from 4 to 17, and Goal-Step's K is not stated.
  • Similarity temperature kappa = not reported
    Controls the sharpness of the feature similarity matrix S in Sec 3.1; no value or sensitivity analysis is given.
  • Positive window alpha = 1
    Ablated in Table 9; determines which narrations are positive samples in Lvna.
  • Negative window beta = all
    Ablated in Table 9; using all non-positive intra-video narrations as negatives was best.
  • Context window Delta = 4
    Ablated in Table 8; extra temporal context used for EgoMCQ evaluation.
  • Contrastive temperature tau = 0.05
    Set in appendix for Lvna and Lft; controls logit sharpness.
  • Temporal edge threshold d = 1
    Graph edges connect segments within temporal distance 1; affects message passing locality.
  • Cut&Match subsample size = not specified
    Spectral clustering is computed on a fixed-size subsample of each graph before 1-NN propagation; the subsample size is not reported.
assumptions (5)
  • domain assumption Spectral clustering of a feature-similarity graph recovers functional threads that correspond to meaningful activity groups.
    Sec 3.1 and Sec 3.3 equate strongly connected graph regions with procedure steps; the mapping from graph clusters to semantic activity threads is assumed.
  • domain assumption Narrations within a temporal window are semantically associated with the corresponding video segments.
    The Lvna loss (Eq. 5) defines positives by timestamp proximity; noisy, sparse, or misaligned narrations would corrupt the learned similarity space.
  • domain assumption Frequently co-occurring actions in unscripted videos are functionally dependent.
    Sec 3 states functional dependencies emerge as frequent co-occurrence patterns; this is a prior about the structure of human activity.
  • ad hoc to paper Subsampling the video graph and propagating cluster labels by 1-NN preserves the full-graph clustering structure.
    Appendix B.1 approximates the eigendecomposition and clustering on a subsample; no bound or empirical validation of this approximation is provided.
  • domain assumption The hierarchical Graph U-Net architecture matches the natural abstraction levels of human behavior.
    The multi-scale coarsening in the encoder and decoder is presented as an inductive bias; the paper does not test alternative architectures without this hierarchy.
invented entities (1)
  • Functional threads
    purpose: A conceptual intermediate structure grouping video segments that are functionally similar, used to enrich features and to detect procedure steps by clustering.
    The construct is defined and detected by the paper's own Cut&Match clustering; it has no independent measurement or falsifiable handle outside the paper's tasks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos." pith.science (2026). https://pith.science/paper/IO6FMK2Y

@misc{pith2026250512911,
  author       = {Pith},
  title        = {Pith review of: HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentric videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IO6FMK2Y}},
  note         = {Machine review of arXiv:2505.12911}
}
read the original abstract

Human activities are particularly complex and variable, and this makes challenging for deep learning models to reason about them. However, we note that such variability does have an underlying structure, composed of a hierarchy of patterns of related actions. We argue that such structure can emerge naturally from unscripted videos of human activities, and can be leveraged to better reason about their content. We present HiERO, a weakly-supervised method to enrich video segments features with the corresponding hierarchical activity threads. By aligning video clips with their narrated descriptions, HiERO infers contextual, semantic and temporal reasoning with an hierarchical architecture. We prove the potential of our enriched features with multiple video-text alignment benchmarks (EgoMCQ, EgoNLQ) with minimal additional training, and in zero-shot for procedure learning tasks (EgoProceL and Ego4D Goal-Step). Notably, HiERO achieves state-of-the-art performance in all the benchmarks, and for procedure learning tasks it outperforms fully-supervised methods by a large margin (+12.5% F1 on EgoProceL) in zero shot. Our results prove the relevance of using knowledge of the hierarchy of human activities for multiple reasoning tasks in egocentric vision.

Figures

Figures reproduced from arXiv: 2505.12911 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Emergence of step clusters in the features similarity matrix of a video from Ego4D [17]. Colored rectangles indicate the ground truth steps. Ideally, we expect high similarity (brighter regions) if two segments represent the same or semantically sim￾ilar steps, e.g. cut onion and cut carrot. On Omnivore features, this behavior is only partially visible. On EgoVLP features, we observe sharper clusters of temporal seg… view at source ↗
Figure 3
Figure 3. Architecture of HiERO. HiERO is designed as an encoder-decoder architecture to implement Function-Aware video-text align￾ment. The Temporal Encoder E performs temporal reasoning on graph representations of the input video at different scales, while the Function-Aware Decoder D recombines nodes in the video graph by matching segments that represent functional dependencies between the actions (Cut & Match module). HiE… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Zero-Shot Localization results on Ego4D Goal-Step, showing some of the HiERO’s success and failure cases. We observe that many failure cases of HiERO are related to the ambiguous granularity of the step annotations in the dataset. In Fig. 4a, HiERO confuses the step Co…
Figure 5
Figure 5. Figure 5: Features distribution of narrations and procedural steps in Goal-Step [48]. Dots and stars represent the textual em￾beddings of the narrations and key-step labels, respectively, while the colors indicate the step to which the narrations are assigned. tion (Fig. 5b), we…
Figure 6
Figure 6. Figure 6: Features distribution of Omnivore and OPEL on MECCANO [41], with dots representing different video segments, and colors encoding the ground truth step labels. Despite not being trained on MECCANO, Omnivore features show a quite distinct separation between segments of t…
Figure 7
Figure 7. Figure 7: Failure cases on the Zero-Shot Localization task on Goal-Step [48], showing the ground truth steps, the predictions obtained by clustering the EgoVLP and HiERO features and the middle frame of each step from the ground truth. We find that most cases of failure are rela…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 59 canonical work pages

  1. [1]

    Ht-step: Aligning instructional articles with how-to videos

    Triantafyllos Afouras, Effrosyni Mavroudi, Tushar Nagara- jan, Huiyu Wang, and Lorenzo Torresani. Ht-step: Aligning instructional articles with how-to videos. In NeurIPS, 2023. 2

    work page 2023

  2. [2]

    Hiervl: Learning hierarchical video- language embeddings

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video- language embeddings. In CVPR, 2023. 1, 2, 6

    work page 2023

  3. [3]

    Video-mined task graphs for keystep recognition in instructional videos

    Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Tri- antafyllos Afouras, and Kristen Grauman. Video-mined task graphs for keystep recognition in instructional videos. NeurIPS, 2023. 2

    work page 2023

  4. [4]

    My view is the best view: Procedure learning from egocentric videos

    Siddhant Bansal, Chetan Arora, and CV Jawahar. My view is the best view: Procedure learning from egocentric videos. In ECCV, 2022. 1, 2, 6, 7, 8, 12, 13, 14, 15

    work page 2022

  5. [5]

    United we stand, divided we fall: Unitygraph for unsupervised proce- dure learning from videos

    Siddhant Bansal, Chetan Arora, and CV Jawahar. United we stand, divided we fall: Unitygraph for unsupervised proce- dure learning from videos. In WACV, 2024. 2, 7, 13

    work page 2024

  6. [6]

    Soft-nms–improving object detection with one line of code

    Navaneeth Bodla, Bharat Singh, Rama Chellappa, and Larry S Davis. Soft-nms–improving object detection with one line of code. In ICCV, 2017. 8

    work page 2017

  7. [7]

    Hier- archically organized behavior and its neural foundations: A reinforcement learning perspective

    Matthew M Botvinick, Yael Niv, and Andew G Barto. Hier- archically organized behavior and its neural foundations: A reinforcement learning perspective. Cognition, 2009. 1

    work page 2009

  8. [8]

    Opel: Optimal transport guided procedure learning

    Sayeed Shafayet Chowdhury, Soumyadeep Chandra, and Kaushik Roy. Opel: Optimal transport guided procedure learning. In NeurIPS, 2024. 1, 2, 7, 13, 15

    work page 2024

Show all 62 references
  1. [9]

    Hierarchical schemas and goals in the control of sequential behavior

    Richard P Cooper and Tim Shallice. Hierarchical schemas and goals in the control of sequential behavior. Psychologi- cal Review, 2006. 1

    2006

  2. [10]

    Carnegie mellon university multimodal activity (cmu-mmac) database, 2008

    Fernando De la Torre, Jessica Hodgins, J Montano, S Val- carcel, R Forcada, and J Macey. Carnegie mellon university multimodal activity (cmu-mmac) database, 2008. 7, 12, 13, 15

    2008

  3. [11]

    Step- former: Self-supervised step discovery and localization in instructional videos

    Nikita Dvornik, Isma Hadji, Ran Zhang, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. Step- former: Self-supervised step discovery and localization in instructional videos. In CVPR, 2023. 2

    2023

  4. [12]

    Self-supervised multi- task procedure learning from instructional videos

    Ehsan Elhamifar and Dat Huynh. Self-supervised multi- task procedure learning from instructional videos. In ECCV,

  5. [13]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019. 6

    2019

  6. [14]

    Graph u-nets

    Hongyang Gao and Shuiwang Ji. Graph u-nets. In ICML,

  7. [15]

    Omnivore: A sin- gle model for many visual modalities

    Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A sin- gle model for many visual modalities. In CVPR, 2022. 1, 6, 8

    2022

  8. [16]

    Amego: Active memory from long egocentric videos

    Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. Amego: Active memory from long egocentric videos. In ECCV, 2024. 2

    2024

  9. [17]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In CVPR, 2022. 3, 6, 7, 14

    2022

  10. [18]

    LoRA: Low-rank adaptation of large language models

    Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 13

    2022

  11. [19]

    Improving action segmentation via graph-based temporal reasoning

    Yifei Huang, Yusuke Sugano, and Yoichi Sato. Improving action segmentation via graph-based temporal reasoning. In CVPR, 2020. 4

    2020

  12. [20]

    Video re- cap: Recursive captioning of hour-long videos

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Na- garajan, Lorenzo Torresani, and Gedas Bertasius. Video re- cap: Recursive captioning of hour-long videos. In CVPR,

  13. [21]

    Epic- tent: An egocentric video dataset for camping tent assembly

    Youngkyoon Jang, Brian Sullivan, Casimir Ludwig, Iain Gilchrist, Dima Damen, and Walterio Mayol-Cuevas. Epic- tent: An egocentric video dataset for camping tent assembly. In ICCVW, 2019. 7, 12, 13, 15

    2019

  14. [22]

    Egotaskqa: Understanding human tasks in egocentric videos

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. In NeurIPS, 2022. 2

    2022

  15. [23]

    Quantifying and learning static vs

    Matthew Kowal, Mennatullah Siam, Md Amirul Islam, Neil DB Bruce, Richard P Wildes, and Konstantinos G Der- panis. Quantifying and learning static vs. dynamic informa- tion in deep spatiotemporal networks. IEEE TPAMI, 2024. 2 9

    2024

  16. [24]

    Kuehne, A

    H. Kuehne, A. B. Arslan, and T. Serre. The language of ac- tions: Recovering the syntax and semantics of goal-directed human activities. In CVPR, 2014. 3

    2014

  17. [25]

    The hungarian method for the assignment problem

    Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 1955. 7

    1955

  18. [26]

    Unsupervised action segmentation by joint representation learning and on- line clustering

    Sateesh Kumar, Sanjay Haresh, Awais Ahmed, Andrey Konin, M Zeeshan Zia, and Quoc-Huy Tran. Unsupervised action segmentation by joint representation learning and on- line clustering. In CVPR, 2022. 2

    2022

  19. [27]

    Set-constrained viterbi for set- supervised action segmentation

    Jun Li and Sinisa Todorovic. Set-constrained viterbi for set- supervised action segmentation. In CVPR, 2020. 2

    2020

  20. [28]

    In the eye of beholder: Joint learning of gaze and actions in first person video

    Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In ECCV, 2018. 7, 12, 13, 15

    2018

  21. [29]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z XU, Difei Gao, Rong-Cheng Tu, Wen- zhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. In NeurIPS, 2022. 2, 4, 5, 6, 7, 8, 12, 13, 14, 15

    2022

  22. [30]

    Learning to recognize procedural activities with distant supervision

    Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. In CVPR, 2022. 2

    2022

  23. [31]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. In NeurIPS, 2023. 2

    2023

  24. [32]

    Learning to ground instructional articles in videos through narrations

    Effrosyni Mavroudi, Triantafyllos Afouras, and Lorenzo Torresani. Learning to ground instructional articles in videos through narrations. In ICCV, 2023. 2

    2023

  25. [33]

    Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization

    Luke Melas-Kyriazi, Christian Rupprecht, Iro Laina, and Andrea Vedaldi. Deep spectral methods: A surprisingly strong baseline for unsupervised semantic segmentation and localization. In CVPR, 2022. 2

    2022

  26. [34]

    Grounded human-object interaction hotspots from video

    Tushar Nagarajan, Christoph Feichtenhofer, and Kristen Grauman. Grounded human-object interaction hotspots from video. In ICCV, 2019. 2

    2019

  27. [35]

    Ego-topo: Environment affordances from egocentric video

    Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In CVPR, 2020. 2

    2020

  28. [36]

    Procedure completion by learning from partial summaries

    Zwe Naing and Ehsan Elhamifar. Procedure completion by learning from partial summaries. In BMVC, 2020. 2

    2020

  29. [37]

    A backpack full of skills: Ego- centric video understanding with diverse task perspectives

    Simone Alberto Peirone, Francesca Pistilli, Antonio Al- liegro, and Giuseppe Averta. A backpack full of skills: Ego- centric video understanding with diverse task perspectives. In CVPR, 2024. 2

    2024

  30. [38]

    Hier- egopack: Hierarchical egocentric video understanding with diverse task perspectives

    Simone Alberto Peirone, Francesca Pistilli, Antonio Al- liegro, Tatiana Tommasi, and Giuseppe Averta. Hier- egopack: Hierarchical egocentric video understanding with diverse task perspectives. arXiv preprint arXiv:2502.02487,

    arXiv

  31. [39]

    Egovlpv2: Egocentric video-language pre-training with fusion in the backbone

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In ICCV, 2023. 1, 2, 5, 6, 7, 12

    2023

  32. [40]

    Unweavenet: Unweaving activity stories

    Will Price, Carl V ondrick, and Dima Damen. Unweavenet: Unweaving activity stories. In CVPR, 2022. 1, 2

    2022

  33. [41]

    Meccano: A multimodal egocentric dataset for hu- mans behavior understanding in the industrial-like domain

    Francesco Ragusa, Antonino Furnari, and Giovanni Maria Farinella. Meccano: A multimodal egocentric dataset for hu- mans behavior understanding in the industrial-like domain. CVIU, 2023. 7, 12, 13, 15, 16

    2023

  34. [42]

    Neuralnetwork-viterbi: A framework for weakly supervised video learning

    Alexander Richard, Hilde Kuehne, Ahsan Iqbal, and Juer- gen Gall. Neuralnetwork-viterbi: A framework for weakly supervised video learning. In CVPR, 2018. 2

    2018

  35. [43]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    V Sanh. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. 6, 13

    1910 arXiv

  36. [44]

    Temporally-weighted hierarchical clustering for unsupervised action segmentation

    Saquib Sarfraz, Naila Murray, Vivek Sharma, Ali Diba, Luc Van Gool, and Rainer Stiefelhagen. Temporally-weighted hierarchical clustering for unsupervised action segmentation. In CVPR, 2021. 1, 2, 7

    2021

  37. [45]

    Differentiable task graph learning: Procedural ac- tivity representation and online mistake detection from ego- centric videos

    Luigi Seminara, Giovanni Maria Farinella, and Antonino Furnari. Differentiable task graph learning: Procedural ac- tivity representation and online mistake detection from ego- centric videos. In NeurIPS, 2024. 2, 3

    2024

  38. [46]

    Progress-aware online ac- tion segmentation for egocentric procedural task videos

    Yuhan Shen and Ehsan Elhamifar. Progress-aware online ac- tion segmentation for egocentric procedural task videos. In CVPR, 2024. 2

    2024

  39. [47]

    Normalized cuts and image segmentation

    Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE TPAMI, 2000. 2

    2000

  40. [48]

    Ego4d goal-step: To- ward hierarchical understanding of procedural activities

    Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: To- ward hierarchical understanding of procedural activities. In NeurIPS, 2024. 1, 2, 6, 7, 8, 12, 14, 15, 17

    2024

  41. [49]

    Combining em- bedded accelerometers with computer vision for recognizing food preparation activities

    Sebastian Stein and Stephen J McKenna. Combining em- bedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, 2013. 3

    2013

  42. [50]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training

    Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. In NeurIPS, 2022. 1

    2022

  43. [51]

    A tutorial on spectral clustering.Statis- tics and computing, 17:395–416, 2007

    Ulrike V on Luxburg. A tutorial on spectral clustering.Statis- tics and computing, 17:395–416, 2007. 3

    2007

  44. [52]

    Ego-only: Egocentric action detection without exocentric transferring

    Huiyu Wang, Mitesh Kumar Singh, and Lorenzo Torresani. Ego-only: Egocentric action detection without exocentric transferring. In ICCV, 2023. 8

    2023

  45. [53]

    Videocutler: Surprisingly simple unsuper- vised video instance segmentation

    Xudong Wang, Ishan Misra, Ziyun Zeng, Rohit Girdhar, and Trevor Darrell. Videocutler: Surprisingly simple unsuper- vised video instance segmentation. In CVPR, 2024. 2

    2024

  46. [54]

    Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut

    Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufrey- daz. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. IEEE TPAMI, 2023. 2

    2023

  47. [55]

    Actionformer: Localizing moments of actions with transformers

    Chen-Lin Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In ECCV,

  48. [56]

    Span-based localizing network for natural language video lo- calization

    Hao Zhang, Aixin Sun, Wei Jing, and Joey Tianyi Zhou. Span-based localizing network for natural language video lo- calization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , 2020. 6, 7, 12, 14 10

    2020

  49. [57]

    Learning video representations from large language models

    Yue Zhao, Ishan Misra, Philipp Kr ¨ahenb¨uhl, and Rohit Gird- har. Learning video representations from large language models. In CVPR, 2023. 1, 2, 5, 6, 7, 12, 13, 14

    2023

  50. [58]

    Learning procedure-aware video rep- resentation from instructional videos and their narrations

    Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xuet- ing Yan, and Yin Li. Learning procedure-aware video rep- resentation from instructional videos and their narrations. In CVPR, 2023. 2

    2023

  51. [59]

    Procedure-aware pretraining for instructional video understanding

    Honglu Zhou, Roberto Mart ´ın-Mart´ın, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Niebles. Procedure-aware pretraining for instructional video understanding. In CVPR,

  52. [60]

    Towards automatic learning of procedures from web instructional videos

    Luowei Zhou, Chenliang Xu, and Jason Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, 2018. 2

    2018

  53. [61]

    #C C stirs food in a frying pan with a spoon in his right hand

    Dimitri Zhukov, Jean-Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivic. Cross- task weakly supervised learning from instructional videos. In CVPR, 2019. 2 11 HiERO: understanding the hierarchy of human behavior enhances reasoning on egocentri...

    2019

  54. [62]

    Bake or roast ingredients in oven

    and Omnivore, and (ii) different temporal contexts used for feature extraction. We replace the ResNet-50 backbone in OPEL with Omnivore, varying the temporal context and stride used for features extraction (Table 13). The two back- bones show comparable performance, with an im...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.