Pith. sign in

REVIEW 3 major objections 5 minor 54 references

EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a single-stage transfer of an image-text CLIP model into a video encoder, with a full-dimension spatial-temporal rotary positional embedding and the SMS loss, reaches state-of-the-art egocentric video-language…

desk verdict The SMS loss ablation is the real contribution; the spatial-temporal RoPE gains outrun the controlled evidence. read the letter →

arxiv 2506.14356 v1 pith:SBRAKSJX submitted 2025-06-17 cs.CV cs.AI

Xiaoqi Wang , Yi Wang , Lap-Pui Chau This is my paper
classification cs.CVcs.AI
keywords egocentricvideounderstandingvideo-languagepretrainingrotarypositionalembeddingsspatial-temporalmodelingmulti-instanceretrievalsymmetricmulti-similaritylosscontrastivelearningEVA02-CLIP
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that egocentric video-language understanding does not need expensive multi-stage pretraining or dimension-split positional encodings. It claims that a single-stage transfer of a pretrained image-text CLIP model into a video encoder, using a full-dimension spatial-temporal rotary positional embedding inside joint attention, reaches state-of-the-art results on Ego4D, EK-100, and Charades-Ego with fewer parameters than predecessors. A second claim is that the Symmetric Multi-Similarity (SMS) loss, which uses soft relevance scores for both positive and negative pairs, is a better objective for multi-instance retrieval than the adaptive MI-MM loss. A sympathetic reader would care because the recipe points to cheaper egocentric video pretraining and a loss that may transfer to other soft-label retrieval problems.

What carries the argument

The load-bearing object is the spatial-temporal rotary positional embedding: with 1D-RoPE over time and 2D-RoPE over space, both spanning the full feature dimension $D$, the composition $R(xy+t)=R_{xy}\cdot R_t$ encodes relative position across axes by rotation-angle addition. It runs inside joint self-attention over all $T\times p^2$ patches, replacing the divided dimensional slices used by 3D-RoPE. The second piece is the SMS loss, which reconstructs a $B\times B$ relevance matrix from soft labels during training and applies a symmetric margin objective with a relaxation factor $\tau$ so that nearly equal pairs stop being optimized.

What would settle it

Train the same EVA02-AT architecture starting from the same CLIP weights used for the AVION baseline, or train vanilla ViT starting from EVA-02-CLIP, and compare on zero-shot EK-100 MIR; if the margin over AVION collapses or reverses, the spatial-temporal RoPE and SMS loss are not the source of the reported improvement.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that RoPE's multiplicative property lets spatial and temporal rotary embeddings be combined over the entire hidden dimension, so a video patch's position is encoded as the sum of a 2D spatial rotation and a 1D temporal rotation rather than as independent slices of the feature vector. This integrated spatial-temporal RoPE, paired with joint attention over all patches, gives the model cross-axis positional relationships without changing parameter count or requiring a divided-attention video backbone. The paper further claims that the SMS loss, built from a batch-wise relevance matrix of verb-noun overlap, symmetrically optimizes positive and negative pairs and consistently outperforms prior soft-label objectives on the EK-100 multi-instance retrieval benchmark.

Load-bearing premise

The load-bearing premise is that the reported gains come from the spatial-temporal RoPE and SMS loss rather than from the stronger EVA-02-CLIP initialization; the paper compares EVA02-AT against baselines initialized from a different CLIP checkpoint, and its own Table IV shows EVA02-AT with only learnable temporal embeddings is slightly worse than vanilla ViT.

Editorial extensions

If this is right

  • If the central claims hold, egocentric video-language models can be built by one-stage transfer from an image-text CLIP checkpoint, cutting pretraining cost compared with three-stage pipelines.
  • The full-dimension spatial-temporal RoPE should let image encoders become video encoders without additional temporal modules, keeping parameter counts fixed.
  • SMS loss should improve multi-instance retrieval whenever soft relevance labels exist, not only on EK-100.
  • On Ego4D EgoMCQ, Charades-Ego, and EK-100, the reported numbers imply that EVA02-AT is the best published egocentric video-language model at matched parameter scale.
  • The zero-shot gains suggest that video-text alignment alone captures enough egocentric semantics for action recognition without fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the SMS loss is a general recipe for any soft-label retrieval setting; if relevance scores are available between all pairs in a batch, the same symmetric margin could apply to image-text, audio-text, or multi-label classification tasks, though the paper only tests egocentric video.
  • Editorial inference: full-dimension composition of spatial and temporal RoPE could be dropped into other video transformers that already use joint attention, such as those built on vanilla ViT, without retraining from scratch; the paper's zero-shot comparisons hint at this generality, but the confounded initialization leaves it untested.
  • Editorial inference: the relaxation factor $\tau$ acts as a soft threshold on correlation differences, so a natural testable extension would be to schedule or learn $\tau$ per batch, since the paper fixes it at 0.1.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. This paper presents EVA02-AT, a video-language model for egocentric understanding. It makes three proposals: a single-stage transfer of the EVA-02-CLIP image encoder to video; a spatial-temporal rotary positional embedding applied over the full hidden dimension that is combined with learnable positional embeddings and joint attention; and a Symmetric Multi-Similarity (SMS) loss for soft-label multi-instance retrieval that uses correlations of both positive and negative pairs. The model is evaluated on Ego4D (EgoMCQ), EPIC-Kitchens-100 MIR, and Charades-Ego in zero-shot and fine-tuned settings, and compared against AVION, EgoVLP, LaViLa, and others. The paper reports state-of-the-art results with parameter counts equal to AVION.

Significance. If the claims hold, the work provides a simple and efficient recipe for converting an image-based CLIP model into a strong egocentric video encoder, and a loss that improves multi-instance retrieval under soft labels. The SMS loss ablation in Table V is well controlled (same backbone, same optimizer) and shows consistent gains over adaptive MI-MM, which is the strongest evidence in the paper. The architecture ablation in Table IV shows that the proposed RoPE contributes +2.0 mAP on the EVA02-AT backbone. The main weakness is that the architecture contribution is not fully separated from the choice of EVA-02-CLIP initialization, leaving a specific confound to be resolved. The public code and models are an additional strength that supports reproducibility.

major comments (3)
  1. [§V-C, Table IV and Table III] The central claim that the EVA02-AT architecture's gains come from the spatial-temporal RoPE is not fully supported because no same-initialization cross-architecture run is reported. Table IV shows that EVA02-AT with learnable temporal PE (28.2 mAP) is statistically tied with vanilla ViT with learnable PE (28.4 mAP), and that adding RoPE on EVA02-AT improves to 30.2 mAP. However, it is unknown whether the 2.0 mAP RoPE gain would also appear on a vanilla ViT backbone or only in combination with EVA-02-CLIP's architecture/weights. Please add an experiment that applies the proposed spatial-temporal RoPE to the vanilla ViT (e.g., AVION) with the same initialization protocol, or that initializes EVA02-AT from OpenAI CLIP, so that the comparison 'EVA02-AT beats ViT-B and ViT-L by 1.4% and 1.3%' in §V-C is not confounded by initialization.
  2. [§IV-B, Eq. (16)] The SMS loss, which is a core contribution, is not defined rigorously enough to reproduce. The set N of triplets is never formally defined; the paper only says that a B×B relevancy matrix is rebuilt during the loss calculation. It is also unclear how the cases R ≥ λ, R ≤ −λ, and |R| < λ in Eq. (16) are derived from Eq. (15), and the notation ‖S_ij − S_ik‖_1 for scalar similarities is nonstandard. Please provide a precise definition of the sampling procedure for (i,j,k), the construction of the batch-wise relevancy matrix, and the exact form of the loss in each case.
  3. [§V-B, Table II] Several zero-shot claims in Table II rely on differences of 0.4–1.0 mAP (e.g., EVA02-AT vs AVION on CharadesEgo zero-shot and on EgoMCQ intra-video). No variance or number of seeds is reported for any experiment, so it is difficult to judge whether these small differences are significant. The large gains in Table I and Table V are more convincing, but the zero-shot architecture claim would be strengthened by reporting mean ± std over at least 3 runs or by noting statistical significance.
minor comments (5)
  1. [Fig. 5 caption] The caption says 'SMS decades more sharply'; this should read 'decays more sharply'.
  2. [Table II] The EVA02-AT rows under the EgoClip and EgoClip+ groups are not labeled as base or large, making the table hard to read; the text reports 'base' and 'large' numbers without indicating which rows they refer to.
  3. [Eq. (7)] The set-builder notation for the positional embeddings uses indices i and j that do not appear in the set members; rewriting this as an explicit replication operation would improve clarity.
  4. [First page] The arXiv version contains unicode artifacts (e.g., '/uni00000028/...') on the first page; these should be cleaned before publication.
  5. [§V-C] In the sentence about the learnable positional embedding gain, '1.4$ gain' contains a stray dollar sign and should be '1.4% gain'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the architecture and loss are explicit constructions and all headline numbers are empirical benchmark evaluations, not derivations from the method's own outputs.

full rationale

The paper contains no load-bearing circular step. The spatial-temporal RoPE is an explicit construction (Eqs. 8-10) that combines 2D spatial and 1D temporal RoPEs over the full hidden dimension via the multiplicative property of RoPE; it is not derived from the benchmarks it later predicts. The SMS loss (Eqs. 14-16) is a piecewise objective defined in terms of soft labels c_ij and similarities S_ij, and the reported EK-100, Charades-Ego, and EgoMCQ results are measured on external evaluation sets after training or fine-tuning. Hyperparameters such as gamma and tau are selected by validation performance (Fig. 6), which is ordinary model selection rather than circularity, because the final numbers are not computed from those validation curves by construction. The comparison of EVA02-AT against AVION and other baselines does raise a genuine architecture-versus-initialization confound: EVA02-AT is initialized from EVA-02-CLIP while most ViT baselines use OpenAI CLIP, and no same-initialization cross-architecture run is provided. However, this is a validity threat to the attribution of gains, not a logical reduction of the method's output to its input. The paper also does not rely on self-citations for its central claims: the EVA-02-CLIP backbone is an external pretrained model, and the cited loss and RoPE works are prior independent contributions. No predicted quantity is defined as the fitted target, no uniqueness theorem is imported from the authors' own prior work, and no known result is merely renamed. Therefore the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The method rests on standard RoPE algebra, soft-label generation from noun/verb IOU, and training hyperparameters tuned on the target benchmarks. No new physical or conceptual entities are introduced. The main free choices are loss margins, thresholds, and pretraining temperatures. The architectural claim rests on an ad-hoc assumption about composing rotations on the full hidden dimension, and the empirical attribution is confounded by the choice of backbone initialization.

free parameters (5)
  • SMS margin gamma = 0.6 for EK-100, 0.3 for Charades-Ego
    Scales the target separation between positive and negative pairs in the SMS loss; selected by validation mAP scanning in Fig. 6.
  • SMS relaxation factor tau = 0.1
    Stops optimizing pairs with nearly equal soft-label correlations; ablation shows 1.8 mAP drop when tau=0.
  • Positive mining threshold lambda/epsilon = 0.1 for EgoVLP-style positive selection
    Controls which partially matched pairs are treated as positives and shapes the batch-wise relevance matrix.
  • POS soft-label weights alpha_p = 0.5 verb, 0.5 noun
    Weights in Eq. 5 define the relevance labels that the SMS loss optimizes; inherited from Wray et al.
  • InfoNCE temperature = 0.05
    Scales logits during Ego4D pretraining; a standard but hand-chosen hyperparameter.
assumptions (5)
  • standard math RoPE is multiplicative: the inner product of two RoPE embeddings depends on relative position, and rotation angles add.
    Invoked through Eq. 1 and Eq. 8 to justify composing spatial and temporal rotations.
  • ad hoc to paper Spatial and temporal rotations can be composed on the same full hidden dimension to produce a meaningful 3D position encoding.
    Eq. 9 defines R(xy+t)=R_xy*R_t; no proof that the frequency scales are compatible or that cross-axis relationships are represented.
  • domain assumption Soft labels computed from noun/verb IOU (Eq. 5) are a reliable ground-truth relevance for video-text pairs.
    The entire SMS loss and the batch-wise relevance matrix depend on these semantic labels, which are treated as oracle annotations.
  • domain assumption The hard-mining rule c_ij >= epsilon selects effective positive pairs during training.
    The dataloader uses this rule to gather partially matched narrations; inherited from EgoVLP and not independently validated in the paper.
  • domain assumption EVA-02-CLIP image representations can be transferred to video by adding temporal positional embeddings, ST-RoPE, and joint attention, then fine-tuning on EgoClip.
    The single-stage pretraining pipeline relies on this transfer working; the paper offers only empirical evidence, no analysis of when it would fail.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization." pith.science (2026). https://pith.science/paper/SBRAKSJX

@misc{pith2026250614356,
  author       = {Pith},
  title        = {Pith review of: EVA02-AT: Egocentric Video-Language Understanding with Spatial-Temporal Rotary Positional Embeddings and Symmetric Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SBRAKSJX}},
  note         = {Machine review of arXiv:2506.14356}
}
read the original abstract

Egocentric video-language understanding demands both high efficiency and accurate spatial-temporal modeling. Existing approaches face three key challenges: 1) Excessive pre-training cost arising from multi-stage pre-training pipelines, 2) Ineffective spatial-temporal encoding due to manually split 3D rotary positional embeddings that hinder feature interactions, and 3) Imprecise learning objectives in soft-label multi-instance retrieval, which neglect negative pair correlations. In this paper, we introduce EVA02-AT, a suite of EVA02-based video-language foundation models tailored to egocentric video understanding tasks. EVA02-AT first efficiently transfers an image-based CLIP model into a unified video encoder via a single-stage pretraining. Second, instead of applying rotary positional embeddings to isolated dimensions, we introduce spatial-temporal rotary positional embeddings along with joint attention, which can effectively encode both spatial and temporal information on the entire hidden dimension. This joint encoding of spatial-temporal features enables the model to learn cross-axis relationships, which are crucial for accurately modeling motion and interaction in videos. Third, focusing on multi-instance video-language retrieval tasks, we introduce the Symmetric Multi-Similarity (SMS) loss and a novel training framework that advances all soft labels for both positive and negative pairs, providing a more precise learning objective. Extensive experiments on Ego4D, EPIC-Kitchens-100, and Charades-Ego under zero-shot and fine-tuning settings demonstrate that EVA02-AT achieves state-of-the-art performance across diverse egocentric video-language tasks with fewer parameters. Models with our SMS loss also show significant performance gains on multi-instance retrieval benchmarks. Our code and models are publicly available at https://github.com/xqwang14/EVA02-AT .

Figures

Figures reproduced from arXiv: 2506.14356 by the authors.

Figure 1
Figure 1. Our EVA02-AT-L model outperforms the previous state-of-the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the label collection mechanism of adaptive MI-MM loss. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of different video RoPEs. Our method conducts both spatial [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Training framework of EVA02-AT. Given an input video clip [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Training curves for different loss functions. Figure (a) shows the loss [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

54 extracted references · 37 canonical work pages

  1. [1]

    Quo vadis, action recognition? a new model and the kinetics dataset,

    J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” inCVPR, 2017

    work page 2017

  2. [2]

    Video summarization through reinforcement learning with a 3d spatio- temporal u-net,

    T. Liu, Q. Meng, J.-J. Huang, A. Vlontzos, D. Rueckert, and B. Kainz, “Video summarization through reinforcement learning with a 3d spatio- temporal u-net,” IEEE Transactions on Image Processing , vol. 31, pp. 1573–1586, 2022

    work page 2022

  3. [3]

    Deep attention network for egocentric action recognition,

    M. Lu, Z.-N. Li, Y. Wang, and G. Pan, “Deep attention network for egocentric action recognition,”IEEE Transactions on Image Processing, vol. 28, no. 8, pp. 3703–3713, 2019

    work page 2019

  4. [4]

    Training a Large Video Model on a Single Machine in a Day

    Y. Zhao and P. Krähenbühl, “Training a large video model on a single machine in a day,”arXiv preprint arXiv:2309.16669, 2023

    work page Pith review arXiv 2023

  5. [5]

    Internvideo: General video foundation models via gen- erative and discriminative learning,

    Y.Wang,K.Li,Y.Li,Y.He,B.Huang,Z.Zhao,H.Zhang,J.Xu,Y.Liu, Z. Wanget al., “Internvideo: General video foundation models via gen- erative and discriminative learning,”arXiv preprint arXiv:2212.03191 , 2022

    arXiv 2022

  6. [6]

    InternVideo2: Scaling foundation models for multimodal video understanding,

    Y. Wang, K. Li, X. Li, J. Yu, Y. He, C. Wang, G. Chen, B. Pei, Z. Yan, R. Zheng, J. Xu, Z. Wang, Y. Shi, T. Jiang, S. Li, H. Zhang, Y. Huang, Y. Qiao, Y. Wang, and L. Wang, “InternVideo2: Scaling foundation models for multimodal video understanding,”arXiv preprint arXiv:2403.15377, 2024

    arXiv 2024

  7. [7]

    EgoVLPv2: Egocentric video-language pre-training with fusion in the backbone,

    S. Pramanick, Y. Song, S. Nag, K. Q. Lin, H. Shah, M. Z. Shou, R. Chellappa, and P. Zhang, “EgoVLPv2: Egocentric video-language pre-training with fusion in the backbone,” inICCV, 2023, pp. 5285– 5297

    work page 2023

  8. [8]

    Egocentric video-language pretraining,

    K. Q. Lin, J. Wang, M. Soldan, M. Wray, R. Yan, E. Z. XU, D. Gao, R.- C. Tu, W. Zhao, W. Kong, C. Cai, W. HongFa, D. Damen, B. Ghanem, W. Liu, and M. Z. Shou, “Egocentric video-language pretraining,” in NeurIPS, vol. 35, 2022, pp. 7575–7586

    work page 2022

Show all 54 references
  1. [9]

    Improving semantic video retrieval models by training with a relevance-aware online mining strategy,

    A. Falcon, G. Serra, and O. Lanz, “Improving semantic video retrieval models by training with a relevance-aware online mining strategy,” Computer Vision and Image Understanding , 2024

    2024

  2. [10]

    Learning video representations from large language models,

    Y. Zhao, I. Misra, P. Krähenbühl, and R. Girdhar, “Learning video representations from large language models,” inCVPR, 2023, pp. 6586– 6597

    2023

  3. [11]

    Egovideo: Exploring egocentric foundation model and downstream adaptation,

    B. Pei, G. Chen, J. Xu, Y. He, Y. Liu, K. Pan, Y. Huang, Y. Wang, T. Lu, L. Wang et al., “Egovideo: Exploring egocentric foundation model and downstream adaptation,”arXiv preprint arXiv:2406.18070, 2024

    2024 arXiv

  4. [12]

    Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,

    Z. Tong, Y. Song, J. Wang, and L. Wang, “Videomae: Masked autoen- coders are data-efficient learners for self-supervised video pre-training,” NeurIPS, vol. 35, pp. 10078–10093, 2022

    2022

  5. [13]

    Laion-400m: Opendatasetofclip-filtered400millionimage-textpairs,

    C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki, “Laion-400m: Opendatasetofclip-filtered400millionimage-textpairs,” arXiv preprint arXiv:2111.02114, 2021

    2021 arXiv

  6. [14]

    Internvid: A large-scale video-text dataset for multimodal understanding and generation,

    Y. Wang, Y. He, Y. Li, K. Li, J. Yu, X. Ma, X. Li, G. Chen, X. Chen, Y. Wang et al., “Internvid: A large-scale video-text dataset for multimodal understanding and generation,” inICLR, 2023

    2023

  7. [15]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution,

    P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Geet al., “Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution,” arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  8. [16]

    EVA-CLIP: Improved training techniques for clip at scale,

    Q. Sun, Y. Fang, L. Wu, X. Wang, and Y. Cao, “EVA-CLIP: Improved training techniques for clip at scale,”arXiv preprint arXiv:2303.15389, 2023

    2023 arXiv

  9. [17]

    CogVideoX: Text- to-video diffusion models with an expert transformer,

    Z. Yang, J. Teng, W. Zheng, M. Ding, S. Huang, J. Xu, Y. Yang, W. Hong, X. Zhang, G. Feng, D. Yin, X. Gu, Y. Zhang, W. Wang, Y. Cheng, T. Liu, B. Xu, Y. Dong, and J. Tang, “CogVideoX: Text- to-video diffusion models with an expert transformer,”arXiv preprint arXiv:2408.06072, 2024

    2024 arXiv

  10. [18]

    EVA-02: A visual representation for neon genesis,

    Y. Fang, Q. Sun, X. Wang, T. Huang, X. Wang, and Y. Cao, “EVA-02: A visual representation for neon genesis,” arXiv preprint arXiv:2303.11331, 2023

    2023 arXiv

  11. [19]

    Multi- similarity loss with general pair weighting for deep metric learning,

    X. Wang, X. Han, W. Huang, D. Dong, and M. R. Scott, “Multi- similarity loss with general pair weighting for deep metric learning,” in CVPR, 2019, pp. 5022–5030

    2019

  12. [20]

    Ego4d: Around the world in 3,000 hours of egocentric video,

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liuet al., “Ego4d: Around the world in 3,000 hours of egocentric video,” inCVPR, 2022, pp. 18995–19012

    2022

  13. [21]

    Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- 100,

    D. Damen, H. Doughty, G. M. Farinella, A. Furnari, J. Ma, E. Kazakos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens- 100,” IJCV, vol. 130, p. 33–55, 2022

    2022

  14. [22]

    Scaling egocentric vision: The epic-kitchens dataset,

    D. Damen, H. Doughty, G. M. Farinella, S. Fidler, A. Furnari, E. Kaza- kos, D. Moltisanti, J. Munro, T. Perrett, W. Price, and M. Wray, “Scaling egocentric vision: The epic-kitchens dataset,” inECCV, 2018

    2018

  15. [23]

    Charades-ego: A large-scale dataset of paired third and first person videos,

    G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and K. Alahari, “Charades-ego: A large-scale dataset of paired third and first person videos,” arXiv preprint arXiv:1804.09626, 2018

    2018 arXiv

  16. [24]

    Long short-term memory,

    S. Hochreiter and J. Schmidhuber, “Long short-term memory,”Neural Computation, vol. 9, no. 8, pp. 1735–1780, 1997

    1997

  17. [25]

    Is space-time attention all you need for video understanding?

    G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” inICML, 2021

    2021

  18. [26]

    Frozen in time: A joint video and image encoder for end-to-end retrieval,

    M. Bain, A. Nagrani, G. Varol, and A. Zisserman, “Frozen in time: A joint video and image encoder for end-to-end retrieval,” inICCV, 2021, pp. 1728–1738

    2021

  19. [27]

    VideoMAE V2: Scaling video masked autoencoders with dual masking,

    L. Wang, B. Huang, Z. Zhao, Z. Tong, Y. He, Y. Wang, Y. Wang, and Y. Qiao, “VideoMAE V2: Scaling video masked autoencoders with dual masking,” in CVPR, 2023, pp. 14549–14560

    2023

  20. [28]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Ruther- ford, S. Cabi, T. Han, Z. Gong, S. Samangooei, M. Monteiro, J. L. Menick, S. Borgeaud, A. Brock, A. Nematzadeh, S. Sharifzadeh, M. a. Bińkowski, R...

    2022

  21. [29]

    Roformer: En- hanced transformer with rotary position embedding,

    J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: En- hanced transformer with rotary position embedding,”Neurocomputing, vol. 568, p. 127063, 2024

    2024

  22. [30]

    Video-LLaMA: An instruction- tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-LLaMA: An instruction- tuned audio-visual language model for video understanding,” https://arxiv.org/abs/2306.02858, 2023

    2023 arXiv

  23. [31]

    VideoRoPE:Whatmakesforgood video rotary position embedding?

    X. Wei, X. Liu, Y. Zang, X. Dong, P. Zhang, Y. Cao, J. Tong, H. Duan, Q.Guo,J.Wang,X.Qiu,andD.Lin,“VideoRoPE:Whatmakesforgood video rotary position embedding?” https://arxiv.org/abs/2502.05173, 2025

    2025 arXiv

  24. [32]

    Supervised contrastive learn- ing,

    P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y. Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan, “Supervised contrastive learn- ing,” in NeurIPS, vol. 33, 2020, pp. 18661–18673

    2020

  25. [33]

    Parameter-free deep multi-modal clustering with reliable contrastive learning,

    Z. Lou, H. Xue, Y. Wang, C. Zhang, X. Yang, and S. Hu, “Parameter-free deep multi-modal clustering with reliable contrastive learning,” IEEE Transactions on Image Processing , vol. 34, pp. 2628–2640, 2025

    2025

  26. [34]

    Cross-modal contrastive learning network for few-shot action recognition,

    X. Wang, Y. Yan, H.-M. Hu, B. Li, and H. Wang, “Cross-modal contrastive learning network for few-shot action recognition,” IEEE Transactions on Image Processing , vol. 33, pp. 1257–1271, 2024

    2024

  27. [35]

    Representation learning with contrastive predictive coding,

    A. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018

    2018 arXiv

  28. [36]

    End-to-end learning of visual representations from uncurated instructional videos,

    A. Miech, J.-B. Alayrac, L. Smaira, I. Laptev, J. Sivic, and A. Zis- serman, “End-to-end learning of visual representations from uncurated instructional videos,” inCVPR, 2020, pp. 9879–9889

    2020

  29. [37]

    Facenet: A unified embed- ding for face recognition and clustering,

    F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embed- ding for face recognition and clustering,” inCVPR, 2015

    2015

  30. [38]

    Circle loss: A unified perspective of pair similarity optimization,

    Y. Sun, C. Cheng, Y. Zhang, C. Zhang, L. Zheng, Z. Wang, and Y. Wei, “Circle loss: A unified perspective of pair similarity optimization,” in CVPR, 2020, pp. 6398–6407

    2020

  31. [39]

    Relevance-based margin for contrastively-trained video retrieval mod- els,

    A. Falcon, S. Sudhakaran, G. Serra, S. Escalera, and O. Lanz, “Relevance-based margin for contrastively-trained video retrieval mod- els,” in ICMR, 2022, pp. 146–157

    2022

  32. [40]

    Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning,

    R. Wang, D. Chen, Z. Wu, Y. Chen, X. Dai, M. Liu, L. Yuan, and Y. Jiang, “Masked video distillation: Rethinking masked feature modeling for self-supervised video representation learning,” inCVPR, 2023, pp. 6312–6322

    2023

  33. [41]

    Fine-grained action retrieval through multiple parts-of-speech embeddings,

    M. Wray, D. Larlus, G. Csurka, and D. Damen, “Fine-grained action retrieval through multiple parts-of-speech embeddings,” inICCV, 2019, pp. 450–459

    2019

  34. [42]

    On semantic similarity in video retrieval,

    M. Wray, H. Doughty, and D. Damen, “On semantic similarity in video retrieval,” in CVPR, 2021, pp. 3650–3660. IEEE TRANSACTIONS ON IMAGE PROCESSING, MAY 2025 11

    2021

  35. [43]

    Egocentric video-language pretraining @ epic-kitchens-100 multi-instance retrieval challenge 2022,

    K. Q. Lin, A. J. Wang, R. Yan, E. Z. Xu, R. Tu, Y. Zhu, W. Zhao, W. Kong, C. Cai, H. Wang, W. Liu, and M. Z. Shou, “Egocentric video-language pretraining @ epic-kitchens-100 multi-instance retrieval challenge 2022,” arXiv preprint arXiv:2207.01334, 2022

    2022 arXiv

  36. [44]

    Collecting highly parallel data for paraphrase evaluation,

    D. Chen and W. B. Dolan, “Collecting highly parallel data for paraphrase evaluation,” inACL, 2011, pp. 190–200

    2011

  37. [45]

    Epic-fusion: Audio-visual temporal binding for egocentric action recognition,

    E. Kazakos, A. Nagrani, A. Zisserman, and D. Damen, “Epic-fusion: Audio-visual temporal binding for egocentric action recognition,” in ICCV, 2019, pp. 5492–5501

    2019

  38. [46]

    Language models are unsupervised multitask learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskeveret al., “Language models are unsupervised multitask learners,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  39. [47]

    HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips,

    A. Miech, D. Zhukov, J.-B. Alayrac, M. Tapaswi, I. Laptev, and J. Sivic, “HowTo100M: Learning a text-video embedding by watching hundred million narrated video clips,” inICCV, 2019, pp. 2630–2640

    2019

  40. [48]

    Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,

    S. Xie, C. Sun, J. Huang, Z. Tu, and K. Murphy, “Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification,” in ECCV, 2018, pp. 305–321

    2018

  41. [49]

    Learning transferable visual models from natural language supervi- sion,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning transferable visual models from natural language supervi- sion,” in ICML, vol. 139, 2021, pp. 8748–8763

    2021

  42. [50]

    Hiervl: Learning hierarchical video-language embeddings,

    K. Ashutosh, R. Girdhar, L. Torresani, and K. Grauman, “Hiervl: Learning hierarchical video-language embeddings,” inCVPR, 2023, pp. 23066–23078

    2023

  43. [51]

    SViTT-Ego: A sparse video- text transformer for egocentric video,

    H. A. Valdez, K. Min, and S. Tripathi, “SViTT-Ego: A sparse video- text transformer for egocentric video,”arXiv preprint arXiv:2406.09462, 2024

    2024 arXiv

  44. [52]

    Decoupled weight decay regularization

    I. Loshchilov and F. Hutter, “Decoupled weight decay regularization.” in ICLR, 2019

    2019

  45. [53]

    DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter,

    V. Sanh, L. Debut, J. Chaumond, and T. Wolf, “DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  46. [54]

    RoBERTA: A robustly optimized bert pretraining approach,

    Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “RoBERTA: A robustly optimized bert pretraining approach,” arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.