{"id":"dbda2fda-2a51-49a0-b5a4-02351b091389","arxiv_id":"2505.12911","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"HiERO learns hierarchical functional threads from egocentric video-narration pairs and uses clustering of the enriched features to obtain state-of-the-art zero-shot procedure learning results on EgoProceL.","lead":"This paper presents HiERO, a weakly-supervised method that enriches egocentric video features with hierarchical activity threads learned from video and narration pairs, improving video-language alignment and enabling zero-shot procedure step discovery. A generalist might read it because it suggests that the hidden structure of everyday activities, rather than hand-labeled steps, can drive strong video understanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline +12.5% over fully-supervised methods on EgoProceL is unsupported: Table 2 compares only unsupervised baselines under a fixed K=7 protocol, and no supervised baseline or adaptive-K mechanism is provided.","rationale":"The reader's weakest assumption correctly identifies the fixed K=7 as a fragile point: EgoProceL tasks have 4-17 key-steps, and no mechanism estimates K from the video, so the zero-shot procedure-learning result is conditional on an external constant. I agree with that, but I think the more load-bearing problem for the central claim is that the abstract's 'outperforms fully-supervised methods by a large margin (+12.5% F1 on EgoProceL)' has no corresponding experiment: Table 2 lists only unsupervised baselines, and the margin is computed against OPEL, not against any supervised method. This is not an internal inconsistency in the method, but it is an unsupported central claim that would need either a new comparison or a corrected abstract. The fixed-K issue compounds it: even the unsupervised comparison is run under a protocol that cannot recover the true step count for most tasks, so the reported F1 is a function of K as much as of the learned features. I would not reject the paper: the architecture and ablations are plausible, the features may indeed cluster procedurally related segments, and the authors are transparent about the fixed-K protocol in a footnote. But the headline claim should be corrected and the K sensitivity should be reported before the conditional acceptance is finalized. This is why I recommend keeping the reader's CONDITIONAL verdict rather than moving to ACCEPT or REJECT.","tokens_in":21736,"tokens_out":6874,"duration_ms":75240,"concrete_test":"Run the EgoProceL evaluation with K set to each task's true key-step count from Table 6, and separately with K swept from 2 to 20, for HiERO and for the best unsupervised baselines using the same features; also train a standard fully-supervised key-step segmentation head (e.g., ActionFormer or a frame-wise classifier) on EgoProceL with EgoVLP features and evaluate it under the same Hungarian-matching protocol. If HiERO's F1 at the true K is not above the supervised baseline, or if its advantage over unsupervised baselines disappears outside K=7, then the +12.5% over fully-supervised methods and the claim that hierarchical threads drive the gain are not substantiated. If both conditions hold, the fixed-K concern is a minor caveat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that HiERO achieves state-of-the-art results everywhere and outperforms fully-supervised methods by +12.5% F1 on EgoProceL in zero-shot. Table 2, which is the only EgoProceL comparison table, contains no fully-supervised method: CnC, GPL-2D, GPL, and OPEL are all unsupervised or self-supervised procedure-learning methods. The +12.5 margin is exactly 44.5 (Ours, EgoVLP) minus 32.0 (OPEL), so the abstract's 'fully-supervised' comparison is not backed by any table row or citation. Additionally, the table uses a fixed number of predicted key-steps, k=7, 'for a fair comparison' (Sec. 4.1.2), while Table 6 reports true key-step counts from 4 to 17, including MECCANO with 17 key-steps. With K=7, spectral clustering cannot represent a 17-step procedure, so the reported F1 is measured under a constraint that is not derived from video content and may affect HiERO and the baselines differently. For the Goal-Step zero-shot results, K is not reported at all; Appendix B.2 only says spectral clustering from scikit-learn is used, so Tables 3 and 4 are not reproducible. The method may still be useful, but the paper's strongest quantitative claim is not supported by the experiments as presented, and the procedure-learning numbers are entangled with an unstated or fixed cluster-count assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"HiERO is a weakly supervised graph encoder-decoder that enriches egocentric video segment features with hierarchical 'functional threads'. A video is represented as a graph whose nodes are fixed-length segments; a temporal encoder aggregates local context, while a function-aware decoder uses spectral clustering (Cut&Match) to partition nodes into K groups and applies temporal message passing within each group. Training combines a video-narration contrastive loss Lvna and a cluster-contrastive loss Lft, using EgoClip narrations as supervision. The authors evaluate the resulting features on EgoMCQ and EgoNLQ with a learned grounding head, and in zero-shot on EgoProceL procedure learning and Ego4D Goal-Step step grounding and localization. The abstract claims state-of-the-art results across all benchmarks and a +12.5% F1 gain over fully-supervised methods on EgoProceL.","tokens_in":22120,"tokens_out":8549,"duration_ms":89147,"significance":"The underlying idea is interesting and timely: if activity hierarchies can emerge from unscripted video plus narrations, then procedure learning and step grounding could be done without step-level supervision. The paper has real strengths: the training signal is anchored to external narrations, the downstream evaluations are external benchmarks, the approach is relatively cheap (under 20 GPU hours), the code is released, and the ablations (Table 5) and clustering comparisons (Tables 10-11) are useful. The qualitative analysis of failure modes is also candid. However, the headline quantitative claims are substantially overstated relative to the evidence, and the procedure-learning numbers depend on a fixed cluster count that is not adapted to the data. The contribution is defensible after the claims are aligned with the experiments.","major_comments":[{"comment":"The abstract's central quantitative claim, that HiERO 'outperforms fully-supervised methods by a large margin (+12.5% F1 on EgoProceL) in zero shot', is not supported by the experiments. Table 2 compares HiERO with Random, CnC, GPL-2D, GPL, and OPEL, all of which are unsupervised or self-supervised procedure-learning methods; no fully-supervised baseline appears in the table or elsewhere in the paper. The +12.5% figure is exactly the difference between Ours (EgoVLP), 44.5, and OPEL, 32.0, so the comparison is to an unsupervised method, not a fully-supervised one. Please add explicit supervised baselines on EgoProceL or, if none are available under the same protocol, reword the abstract and Section 4.1.2 so that the claim is limited to the methods actually compared.","section":"Abstract; Sec. 4.1.2; Table 2"},{"comment":"All EgoProceL results use a fixed number of clusters K=7, while Table 6 reports that the true number of key-steps in the benchmark ranges from 4 (Greek Salad, Sandwich) to 17 (MECCANO). With K=7, spectral clustering cannot represent a 17-step procedure, so the reported F1 and IoU are measured under a constraint that is not derived from video content and may cap the achievable score on several datasets. Please report per-dataset results with K equal to the true key-step count (or with an estimated K) in addition to the fixed-K protocol, and discuss how the mismatch affects HiERO and the baselines. Similarly, Tables 3 and 4 do not state the K used for zero-shot step grounding and localization; Appendix B.2 only mentions scikit-learn spectral clustering. This makes the Goal-Step zero-shot results irreproducible and leaves open the possibility that an undisclosed K drives the outcome. Please report K and any tuning used for each task.","section":"Sec. 4.1.2; Table 2; Table 6; Appendix B.2"},{"comment":"The abstract's statement that HiERO 'achieves state-of-the-art performance in all the benchmarks' is too broad. In Table 3, zero-shot HiERO (EgoVLP) reaches R@1 11.57 at mIoU@0.3, below the supervised EgoVLP baseline at 15.64; in Table 4, zero-shot HiERO reaches average mAP 8.7, below supervised EgoOnly at 13.6. The paper may intend 'state of the art among zero-shot methods', but that is not what the abstract and conclusions say. Please qualify every SOTA claim by setting (supervised vs zero-shot, fixed-K vs free-K) and by metric, since several metrics in Table 1 also do not improve over the backbone (e.g., EgoNLQ R@5 at mIoU@0.3 decreases from 22.38 to 21.12 with LAVILA features).","section":"Abstract; Sec. 4.1.3; Tables 3 and 4"}],"minor_comments":[{"comment":"The functional-threads loss uses cluster assignments from the model's own Cut&Match module on its own current features. Because the main evaluations and the Lvna anchor are external, I do not see circularity in the evaluation sense, but the paper should briefly discuss potential degenerate solutions (e.g., trivial cluster assignments) and how the combination with Lvna avoids them.","section":"Sec. 3.2.1, Eq. (6)"},{"comment":"The notation for the edge weight w_ij and the sign term s_ij should be clarified: in Eq. (3), w_ij appears to be a scalar computed in Eq. (4), but the expression 'w_ij ⊙ x'_j' uses an elementwise symbol; please reconcile the dimensions or explain the broadcasting.","section":"Sec. 3.1, Eqs. (3)-(4)"},{"comment":"The approximation that propagates cluster labels from a temporally subsampled graph to the original graph by the temporally closest node is a strong assumption, since functional threads need not be temporally contiguous; please validate this approximation or state its expected effect on training.","section":"Appendix B.1"},{"comment":"The differences between spectral clustering and K-Means on EgoProceL are small (e.g., 44.5 vs 44.2 F1 for Ours EgoVLP), and no variance is reported; consider reporting multiple runs or significance in the clustering comparison.","section":"Tables 10 and 11"},{"comment":"There are minor typographical issues: 'LoRa' should be 'LoRA', 'L AVILA' should be 'LAVILA', and the figure labels in Fig. 3 ('Sampling Interp.', 'Func. Thread Generation') do not exactly match the terms defined in the text ('Cut & Match module'); please align them.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript would benefit from a careful pass to distinguish zero-shot from supervised settings and to report per-dataset K values. The current abstract overstates the results, but the core idea and the external anchoring of the training signal are sound; after the claims are corrected and the K-dependence is documented, the paper is likely suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"HiERO is a weakly-supervised hierarchical graph encoder-decoder for egocentric video. The new piece is the full architecture: a temporal encoder that aggregates local context, a function-aware decoder that splits the graph via spectral clustering (Cut&Match), and a contrastive loss that reinforces those clusters. The zero-shot procedure-learning protocol on EgoProceL and Goal-Step is also new, and the results are consistently positive across three backbones (Omnivore, EgoVLP, LaViLa). The appendix is honest and informative: ablations on context window, clustering algorithm, and a real discussion of failure cases. That is solid work.\n\nNow the soft spots. The abstract's headline claim — SOTA everywhere, +12.5% F1 over fully-supervised on EgoProceL — is not supported by the tables. Table 2 compares against unsupervised baselines (CnC, GPL, OPEL); there is no fully-supervised row. The +12.5 is just Ours (EgoVLP) 44.5 minus OPEL 32.0. On Goal-Step, zero-shot is below the supervised EgoOnly baseline (8.7 vs 13.6 mAP). The \"SOTA in all benchmarks\" is also overstated: EgoMCQ gains are about one point, and on EgoNLQ they beat published baselines on only some metrics.\n\nThe fixed-K issue is real. EgoProceL uses k=7 predicted key-steps even though MECCANO has 17 ground-truth steps. That is a fair protocol if all baselines share it, but it caps what HiERO can express, and there is no mechanism to estimate K from video content. K for Goal-Step zero-shot runs is not reported (the appendix just says scikit-learn spectral clustering), which is a reproducibility gap. No error bars anywhere. The functional-threads loss Lft is self-referential, but since the main training signal (Lvna) is anchored to external narrations, I don't see it as a circularity flaw. The paper also claims a project page but ships no code; code release should be a condition.\n\nVerdict: the central idea survives the presentation. The architecture is plausible, the ablations back it, and the failure-case discussion is candid. This deserves a serious referee, but the authors must fix the public claims, disclose K, provide confidence intervals, and release code. Send it to review, and tell the authors to rewrite the abstract.","headline":"A competent hierarchical-graph method with genuinely useful ablations, undermined by the abstract's unsupported claim of beating fully-supervised methods.","tokens_in":22644,"tokens_out":2350,"would_cite":true,"duration_ms":24942,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HiERO shows that hierarchical activity threads, learned from narrations alone, make egocentric video features strong enough to beat supervised procedure learning in zero shot.","keywords":["egocentric video understanding","hierarchical activity structure","procedure learning","weakly-supervised video-text alignment","spectral clustering","zero-shot step localization","graph neural networks","functional threads"],"falsifier":"Re-run the EgoProceL zero-shot experiment with K set to each video's ground-truth number of key-steps (available in Table 6 of the paper) and compare F1 to the fixed-K=7 result; alternatively, sweep K for each video and check whether the F1 peak tracks the true step count. If performance depends on providing the correct K, the claim that steps emerge without task-specific supervision is weakened.","tokens_in":21523,"feed_emoji":"🎬","tokens_out":6891,"duration_ms":70811,"temperature":0.7,"pith_summary":"HiERO argues that human activities in unscripted egocentric video carry a hidden hierarchy: individual actions group into functional threads (e.g., prepping vegetables, cooking pasta), and those threads group into broader routines. The paper claims this hierarchy can be learned without step-level supervision, using only the alignment between short video clips and their narrated descriptions, and that enriching segment features with these multi-scale groupings improves downstream reasoning. On procedure learning, the enriched features allow steps to be recovered by clustering alone: HiERO reports state-of-the-art results in zero shot, outperforming fully-supervised methods by +12.5% F1 on EgoProceL. The same features also improve video-text alignment benchmarks (EgoMCQ, EgoNLQ) with minimal extra training. A sympathetic reader would care because it suggests that the bottleneck for understanding long, unscripted human activity is not more supervision but the right inductive bias about how actions organize.","feed_headline":"Action hierarchy from narrations beats supervised step learning","feed_subtitle":"HiERO clusters egocentric video into functional threads, lifting zero-shot F1 by 12.5 points on EgoProceL.","key_machinery":"The central mechanism is a spectral-clustering-based 'Cut & Match' module inside a Graph U-Net-style encoder-decoder. The video is a temporal graph of segment nodes; the Temporal Encoder aggregates information within local temporal neighborhoods at progressively coarser scales, while the Function-Aware Decoder, at each stage, partitions the current node set into K clusters by spectral clustering on a cosine-similarity graph, runs temporal reasoning inside each cluster separately, then reassembles the graph. This is what makes actions that are temporally distant but functionally related (e.g., 'cut onion' and 'cut carrot') become neighbors in feature space. Training combines a windowed video-narration contrastive loss, which makes co-occurring actions similar, with a functional-thread contrastive loss, which sharpens the clusters discovered by Cut & Match.","core_discovery":"On its own terms, the paper's claim is that hierarchical functional structure is not an annotation to be provided but a pattern that can emerge from data. HiERO models a long video as a graph whose nodes are fixed-length segments, trains an encoder-decoder that propagates information through temporally local neighborhoods while repeatedly cutting the graph into functionally similar clusters and matching nodes across those clusters, and supervises the whole process with a contrastive video-narration alignment loss applied at the decoder output and a contrastive loss that pulls same-cluster segments together. After training, procedure steps are detected by running spectral clustering on the learned segment features, with no task-specific fine-tuning. The paper reports that this zero-shot pipeline outperforms fully-supervised procedure learning baselines on EgoProceL (+12.5% F1), and that the same features improve EgoMCQ intra accuracy and EgoNLQ recall over strong video-text pretraining backbones.","pith_inferences":["Not tested in the paper: the reported zero-shot procedure-learning results fix the cluster count at k=7 on EgoProceL even though benchmark videos contain between 4 and 17 key-steps, so an adaptive estimator of K is a natural extension and the strength of the 'zero-shot' claim depends on whether performance survives when K is not tuned or known.","Not tested in the paper: the step-localization failures shown are mostly granularity mismatches with the annotation taxonomy, so the method appears to discover clusters at its own natural abstraction level; aligning that level with an external step/substep taxonomy is a separate open problem.","Not tested in the paper: because step labels are assigned by cosine similarity between averaged cluster features and taxonomy text embeddings, using a language model to summarize a cluster's narrations before matching could sharpen zero-shot classification without changing the clustering.","Extrapolating beyond the paper: the hierarchy learned from narrations alone could plausibly transfer to non-egocentric video of human activity, since the training signal is co-occurrence of narrated actions rather than camera viewpoint; the paper does not test this."],"forward_implications":["Procedure learning becomes a zero-shot clustering operation: given enough narrated egocentric video, key-steps can be recovered from segment features without a single step-level annotation.","The same learned features transfer across tasks, so a single pretraining pass can support multiple-choice video-text retrieval, natural language temporal grounding, step grounding, and step localization.","Because the method does not need multiple scripted demonstrations of the same procedure, it can be applied to unscripted daily-life video where the same task is never performed twice in the same way.","The improvements hold across several base feature extractors, suggesting that the hierarchical enrichment is complementary to the choice of backbone.","If the zero-shot results are correct, weakly-supervised hierarchy discovery should become a standard component of egocentric video representation learning."],"supporting_citations":[{"why":"Supplies the EgoProceL benchmark, the fixed k=7 evaluation protocol, and the supervised/weakly-supervised baselines that HiERO surpasses in zero-shot procedure learning.","marker":"[4]"},{"why":"Provides the previous state-of-the-art procedure-learning baseline (OPEL) whose F1/IoU numbers are the main comparison on EgoProceL.","marker":"[8]"},{"why":"Supplies the EgoVLP backbone features, the video-narration alignment pretraining approach, and the EgoMCQ benchmark.","marker":"[29]"},{"why":"Supplies the Ego4D Goal-Step step-grounding and step-localization benchmarks, the step taxonomy, and supervised baselines.","marker":"[48]"},{"why":"Provides the Graph U-Net architecture template that HiERO adapts into its temporal encoder and function-aware decoder.","marker":"[14]"},{"why":"Provides the spectral clustering algorithm used by the Cut & Match module to discover functional threads as strongly connected graph regions.","marker":"[51]"},{"why":"Provides the Ego4D dataset, source of the narrated egocentric videos used for training and for the EgoNLQ evaluation.","marker":"[17]"},{"why":"Provides the LaViLa backbone and text encoder, the strongest base features HiERO builds on for EgoMCQ and EgoNLQ.","marker":"[57]"},{"why":"Provides the VSLNet grounding head used to evaluate HiERO features on EgoNLQ and on Goal-Step in supervised settings.","marker":"[56]"}],"fun_headline_variants":["Zero-shot step learning from egocentric hierarchy","Action hierarchy from narrations beats supervised F1","HiERO: hierarchy to reason on egocentric video","Learn hierarchical actions, win zero-shot procedure tasks","Narration-driven hierarchy lifts step F1 by 12.5"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that procedure steps show up as clean clusters in the learned feature space, yet the zero-shot evaluation fixes the cluster count at k=7 on EgoProceL while the benchmark's videos vary from 4 to 17 key-steps; if the true step count differs from K, spectral clustering cannot recover the correct segmentation and no mechanism in HiERO estimates K from the video content.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot step learning from egocentric hierarchy","Action hierarchy from narrations beats supervised F1","HiERO: hierarchy to reason on egocentric video","Learn hierarchical actions, win zero-shot procedure tasks","Narration-driven hierarchy lifts step F1 by 12.5"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1351,"prompt_tokens":950,"completion_tokens":401,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":566,"completion_tokens_details":{"reasoning_tokens":325}},"tokens_in":566,"tokens_out":401,"duration_ms":4311,"temperature":1.0,"reasoning_tokens":325,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:24:25.491089+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the EgoProceL zero-shot experiment with K set to each video's ground-truth number of key-steps (available in Table 6 of the paper) and compare F1 to the fixed-K=7 result; alternatively, sweep K for each video and check whether the F1 peak tracks the true step count. If performance depends on providing the correct K, the claim that steps emerge without task-specific supervision is weakened.","supporting_citations":[{"cited_title":"My view is the best view: Procedure learning from egocentric videos","cited_arxiv_id":null,"evidence_quote":"Supplies the EgoProceL benchmark, the fixed k=7 evaluation protocol, and the supervised/weakly-supervised baselines that HiERO surpasses in zero-shot procedure learning."},{"cited_title":"Opel: Optimal transport guided procedure learning","cited_arxiv_id":null,"evidence_quote":"Provides the previous state-of-the-art procedure-learning baseline (OPEL) whose F1/IoU numbers are the main comparison on EgoProceL."},{"cited_title":"Egocentric video-language pretraining","cited_arxiv_id":null,"evidence_quote":"Supplies the EgoVLP backbone features, the video-narration alignment pretraining approach, and the EgoMCQ benchmark."},{"cited_title":"Ego4d goal-step: To- ward hierarchical understanding of procedural activities","cited_arxiv_id":null,"evidence_quote":"Supplies the Ego4D Goal-Step step-grounding and step-localization benchmarks, the step taxonomy, and supervised baselines."},{"cited_title":"Graph u-nets","cited_arxiv_id":null,"evidence_quote":"Provides the Graph U-Net architecture template that HiERO adapts into its temporal encoder and function-aware decoder."},{"cited_title":"A tutorial on spectral clustering.Statis- tics and computing, 17:395–416, 2007","cited_arxiv_id":null,"evidence_quote":"Provides the spectral clustering algorithm used by the Cut & Match module to discover functional threads as strongly connected graph regions."},{"cited_title":"Ego4d: Around the world in 3,000 hours of egocentric video","cited_arxiv_id":null,"evidence_quote":"Provides the Ego4D dataset, source of the narrated egocentric videos used for training and for the EgoNLQ evaluation."},{"cited_title":"Learning video representations from large language models","cited_arxiv_id":null,"evidence_quote":"Provides the LaViLa backbone and text encoder, the strongest base features HiERO builds on for EgoMCQ and EgoNLQ."},{"cited_title":"Span-based localizing network for natural language video lo- calization","cited_arxiv_id":null,"evidence_quote":"Provides the VSLNet grounding head used to evaluate HiERO features on EgoNLQ and on Goal-Step in supervised settings."}],"review_version":1}