Pith. sign in

REVIEW 3 major objections 5 minor 51 references

AdaVid: Adaptive Video-Language Pretraining

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read AdaVid makes a video encoder's hidden width adjustable at inference, so one pretrained model covers many compute budgets and beats its fixed-width baseline at matched cost.

desk verdict Solid adaptive-width video encoder, but the headline EgoVLP comparison needs a controlled finetuned baseline before it convinces. read the letter →

arxiv 2504.12513 v1 pith:4V5IGLPO submitted 2025-04-16 cs.CV cs.AI

Chaitanya Patel , Juan Carlos Niebles , Ehsan Adeli This is my paper · ORCID
classification cs.CVcs.AI
keywords adaptivecomputationvideo-languagepretrainingtransformerefficiencyMatryoshkarepresentationlearningegocentricvideolongunderstandingembeddingdimensionedgedeployment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that a video-language encoder can be trained so that the same network runs at widely different compute costs by slicing each transformer layer's hidden dimension at inference time. In this AdaVid framework, each layer is trained with randomly chosen widths, so a single model effectively contains many smaller models and needs no postprocessing to switch between them. On egocentric video-text pretraining, AdaVid-EgoVLP matches the standard EgoVLP at roughly half the compute and outperforms it when given equal compute. The same adaptive model can also spend its compute savings on more input frames, which helps on long-video tasks like Diving-48. A lightweight aggregator, AdaVid-Agg, extends the idea to long videos and matches a stronger hierarchical baseline at lower cost.

What carries the argument

The adaptive transformer layer is the mechanism that carries the argument: each linear projection $y = Wx + b$ is restricted to the upper-left $d\times d$ submatrix $W[1:d,1:d]$ and the first $d$ bias entries, layer normalization uses the first $d$ scale and shift values, and multi-head attention uses $d/H$ heads instead of shrinking each head. Randomly sampling widths during pretraining forces a coarse-to-fine structure in the weights and activations, so the trained model can later be evaluated at any allowed width, and the paper finds this acts as a mild regularizer.

What would settle it

Train vanilla EgoVLP with the same recipe used for AdaVid-EgoVLP, meaning the same epoch count, batch size, and EgoVLP initialization but with fixed full width, and compare on EgoMCQ at equal FLOPs. If vanilla finetuning closes the gap, adaptive-width training is not the cause of the reported advantage; a further check is pretraining AdaVid-EgoVLP from scratch and measuring whether it still matches EgoVLP at half compute.

Watch

Extended reading notes

Core claim

The central claim is that the computational cost of a video transformer can be made adjustable without retraining or postprocessing by making the embedding dimension of every layer adaptive. During training the layer width is sampled randomly from the set $\{D, 3D/4, D/2, D/4\}$; at inference the model drops trailing dimensions and uses the corresponding leading submatrices of its weights, with attention using fewer heads rather than narrower heads. AdaVid-EgoVLP, pretrained on Ego4D with the EgoNCE objective, scores 90.8/59.5 on EgoMCQ inter/intra versus 90.6/57.2 for vanilla EgoVLP at full width, and retains most of that accuracy at half compute. On Diving-48, AdaVid-EgoVLP-FT at 128 frames and 285e10 FLOPs reaches 82.2%, ahead of TimeSformer-L at 96 frames and 714e10 FLOPs. The paper also claims that a decreasing-width schedule, with wide early layers and narrow deep layers, works better than the reverse.

Load-bearing premise

The comparison to vanilla EgoVLP assumes that AdaVid-EgoVLP's gains come from adaptive-width training rather than from its extra 10 epochs of finetuning and its initialization from already-trained EgoVLP weights; vanilla EgoVLP receives no equivalent finetuning.

Editorial extensions

If this is right

  • At full width, AdaVid training acts as a regularizer: AdaVid-EgoVLP beats vanilla EgoVLP on EgoMCQ even though it is the same architecture trained with the same supervision.
  • A fixed compute budget can be spent on more frames: AdaVid-EgoVLP-FT reaches 82.2% on Diving-48 with 128 frames and 285e10 FLOPs, where TimeSformer-L needs 714e10 FLOPs for 81.0%.
  • Decreasing width across layers is the better design: decreasing configurations outperform increasing ones at matched FLOPs on EgoMCQ, consistent with deeper layers storing only high-level concepts.
  • Long-video understanding inherits the same flexibility: AdaVid-Agg matches HierVL on SummaryMCQ and EgoSchema at full width and retains most of its accuracy at 0.25x compute.
  • A single AdaVid model gives a smooth compute-accuracy operating curve, from about 0.06x to 1.0x of the reference model's FLOPs, so deployment can pick the width that fits the available device budget.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the slicing mechanism is stated purely in terms of weight submatrices, it should transfer to other transformer-based encoders, including image-language models; the paper only demonstrates it on video.
  • An untested but natural deployment pattern is two-stage retrieval: rank candidates with a narrow width and rerank the top few at full width, an approach the paper mentions as future work.
  • A from-scratch training run would separate the regularization effect from the finetuning effect; if the equal-compute advantage survives, adaptive-width pretraining could become a standard recipe for contrastive video models rather than a special-purpose efficiency trick.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes AdaVid, an adaptive transformer framework that lets a single video encoder be evaluated at multiple hidden widths by slicing nested weight matrices, following the Matryoshka Representation Learning idea. The authors instantiate AdaVid-EgoVLP, an adaptive variant of EgoVLP trained on Ego4D video-narration pairs, and AdaVid-Agg, a lightweight aggregator for long videos. They evaluate on EgoMCQ, Diving-48, SummaryMCQ, LongVideoRetrieval, and EgoSchema, and report that AdaVid-EgoVLP matches or outperforms vanilla EgoVLP at half or equal compute, and that adaptive width lets the model process more frames within a fixed FLOP budget. The paper also studies whether embedding dimensions should decrease or increase across layers during training.

Significance. If the claims were established, the contribution would be practically significant: a single video-language encoder that serves multiple compute budgets at inference time, with no distillation or post-hoc pruning, and with a simple nested-weight design. The within-model compute-accuracy curves across d-768, d-576, d-384, and d-192 are a genuine strength and appear to support graceful accuracy degradation with reduced width. The FLOPs formulas in Appendix A are transparent and easy to check. However, the headline comparison to EgoVLP is not yet controlled, because AdaVid-EgoVLP is initialized from EgoVLP weights and additionally finetuned, while the EgoVLP baseline is not. The Diving-48 comparison with TimeSformer is also confounded by different pretraining data. The adaptive-width idea is plausible and the framework is simple, but the central attribution claim needs a control experiment before the reported advantages can be assigned to the method.

major comments (3)
  1. [Section 4.3, Table 2] The comparison of AdaVid-EgoVLP (90.8/59.5 on EgoMCQ) with vanilla EgoVLP (90.6/57.2) conflates the adaptive-width mechanism with two additional interventions. Section 4.3 states that AdaVid-EgoVLP is initialized from EgoVLP weights and finetuned for 10 epochs with a batch size of 160, whereas the EgoVLP baseline numbers appear to come from the original model without this additional finetuning. To support the abstract's claim that AdaVid "even outperforms EgoVLP when given equal computational resources," the authors must add a control that finetunes vanilla EgoVLP under the identical recipe (same epochs, batch size, initialization, hyperparameters, and EgoNCE loss) while keeping the embedding dimension fixed at 768. If that control reaches 90.8/59.5, the reported advantage is due to extra finetuning rather than to AdaVid; if it does not, the comparison still needs the control to attribute the gain. The statement in Section 3.3 that AdaVid-trained models "often outperform their vanilla counterparts despite identical training setups" is unsupported because no identical-training-setup vanilla model is evaluated.
  2. [Section 4.3, Table 3] The Diving-48 comparison is confounded. The text says TimeSformer shares the same architecture, "ensuring a fair comparison," but the table caption states that the baselines are pretrained on ImageNet-21K while the AdaVid models are pretrained on Ego4D; additionally, AdaVid-EgoVLP-FT starts from EgoVLP weights and is finetuned on Diving-48. Therefore the frame-count-versus-compute claim, for example 128 frames at 285e10 FLOPs reaching 82.2% versus TimeSformer-L at 714e10 FLOPs reaching 81.0%, does not isolate the adaptive-width contribution. At minimum, these results should be reworded as cross-paper comparisons rather than controlled experiments, and an apples-to-apples control with the same pretraining data and finetuning recipe, using a non-adaptive baseline, should be run if the "more frames within limited compute" claim is to be attributed to the method.
  3. [Section 3.3 / Section 4.3] The dimension sampling strategy is underspecified, which blocks reproduction of the dec-versus-inc finding. Section 3.3 says a dimension is "chosen randomly or based on some strategy" from {D, 3D/4, D/2, D/4}, but the paper never specifies the sampling distribution, whether each layer is sampled independently per iteration, how the decreasing or increasing constraint is enforced within a step, or whether the schedule changes over the 10 epochs. The claims that "gradually decreasing embedding dimension sizes across layers yields better performance" and that the opposite strategy "bottlenecks the information at early layers" depend on this exact procedure. Please provide the full sampling algorithm, including the random seed policy, or release code; without this, the central training recipe is not precisely defined.
minor comments (5)
  1. [Section 2] There is a typo in "compute-contrained" which should be "compute-constrained."
  2. [Table 3] The TimeSformer citations are inconsistent: the table cites [2] (Frozen in Time) where [5] (TimeSformer) is presumably intended.
  3. [Section 4.3, Table 2] The source of the EgoVLP baseline numbers should be stated explicitly, namely whether they are taken from the original paper or re-evaluated with the same evaluation code, and confidence intervals or repeated-seed variance for EgoMCQ would help interpret the 0.2-point inter and 2.3-point intra gaps.
  4. [Section 4.2] The LongVideoRetrieval benchmark is newly curated in this paper, but no dataset URL or release instructions are given; please include this information for reproducibility.
  5. [Section 4.4, Table 5] There is a typo in the table caption: "signficantly" should be "significantly."

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity; the EgoVLP-initialization and finetuning choice is a fairness confound for the benchmark comparison, but no derivation reduces to its own inputs.

full rationale

AdaVid is an empirical architecture paper: the adaptive transformer layer is defined constructively (upper-left d x d weight slicing, bias and LayerNorm slicing, head reduction for multi-head attention), and all reported results are measured on fixed external benchmarks (EgoMCQ, Diving-48, SummaryMCQ, EgoSchema, LongVideoRetrieval) rather than produced by fitting an equation to the quantity being reported. The paper's citations to EgoVLP, HierVL, TimeSformer, MRL, FlexViT, and SHARCS are to external prior work with no author overlap, and no load-bearing premise depends on a self-citation. The main self-referential element is experimental: Section 4.3 states "we initialize our model with EgoVLP weights and finetune it with adaptive embedding dimensions" at batch size 160 versus EgoVLP's 512, while the EgoVLP baseline numbers come from the original EgoVLP checkpoint, so the claim that AdaVid-EgoVLP "outperforms EgoVLP" at equal compute conflates initialization, 10 extra epochs, and the adaptive mechanism; Section 3.3's phrase "despite identical training setups" is therefore inaccurate. This is a genuine control deficiency and a correctness risk, but it is not circularity under the rubric: no reported accuracy is equal to, or forced by, an input parameter by construction. The Diving-48 comparison is similarly confounded by different pretraining data (ImageNet-21K versus Ego4D) and by the EgoVLP initialization, but again this is a benchmarking fairness issue rather than a derivation-level circular step.

Assumptions & free parameters 5 free parameters · 6 assumptions · 0 invented entities

The central claims rest on standard transformer FLOPs formulas, the empirical validity of width-dropping after adaptive training, and the transferability of Ego4D pretraining to the chosen benchmarks. No parameters are fitted to a target result; the main free choices are the dimension ratio set, the decreasing-width schedule, and aggregator hyperparameters. The comparison with EgoVLP also implicitly assumes that initialization from EgoVLP plus extra finetuning is a fair protocol.

free parameters (5)
  • Allowed embedding dimension ratios = {1, 0.75, 0.5, 0.25} of D=768
    Hand-chosen set of widths that determines the achievable FLOPs reduction and the granularity of the compute-accuracy trade-off.
  • Dimension scheduling strategy = decreasing (d-dec); selected after comparing variants on EgoMCQ
    The paper compares increasing vs decreasing layer widths and picks the better one after seeing benchmark results; this is a design choice tuned on the evaluation benchmark.
  • AdaVid-Agg segment count and frame count = S=16, T=64
    Architectural choices for the hierarchical aggregator; they affect compute and accuracy on long-video benchmarks.
  • Number of aggregator transformer layers = 12
    Matches HierVL's aggregator capacity; chosen without a systematic search.
  • Training epochs and batch sizes = 10 epochs; batch 160 (AdaVid-EgoVLP), 256 (AdaVid-Agg)
    Standard hyperparameters. The lower batch size relative to EgoVLP (512) is compensated by initialization from EgoVLP weights, which is itself a design choice.
assumptions (6)
  • standard math Space-time attention (TimeSformer/EgoVLP) is a suitable base video encoder, and the complexity formulas in Appendix A are correct.
    The paper's FLOPs accounting assumes standard transformer complexity; used in Tables 1, 3, and Figure 6.
  • domain assumption Dropping trailing dimensions of weights and activations, and halving heads, after adaptive training preserves the semantic information needed for downstream tasks.
    This is the core hypothesis of AdaVid; supported only empirically by Figures 3 through 6, not by any guarantee.
  • domain assumption A coarse-to-fine structure, where deeper layers use smaller embedding dimensions, is beneficial.
    Motivated by SHARCS and the paper's own comparison of d-dec vs d-inc on EgoMCQ (Figure 3); assumed to transfer to other datasets.
  • domain assumption Ego4D narration and summary supervision transfers to EgoMCQ, SummaryMCQ, EgoSchema, Diving-48, and LongVideoRetrieval.
    The method is evaluated zero-shot or finetuned across these benchmarks; transferability is assumed.
  • domain assumption Hierarchical late fusion, encoding short clips and then aggregating, captures long-video semantics.
    Same assumption as HierVL; the paper does not justify it beyond citing prior work.
  • domain assumption FLOPs are an accurate proxy for the computational footprint on edge devices.
    All compute comparisons are in FLOPs; no latency, energy, or memory measurements are reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AdaVid: Adaptive Video-Language Pretraining." pith.science (2026). https://pith.science/paper/4V5IGLPO

@misc{pith2026250412513,
  author       = {Pith},
  title        = {Pith review of: AdaVid: Adaptive Video-Language Pretraining},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4V5IGLPO}},
  note         = {Machine review of arXiv:2504.12513}
}
read the original abstract

Contrastive video-language pretraining has demonstrated great success in learning rich and robust video representations. However, deploying such video encoders on compute-constrained edge devices remains challenging due to their high computational demands. Additionally, existing models are typically trained to process only short video clips, often limited to 4 to 64 frames. In this paper, we introduce AdaVid, a flexible architectural framework designed to learn efficient video encoders that can dynamically adapt their computational footprint based on available resources. At the heart of AdaVid is an adaptive transformer block, inspired by Matryoshka Representation Learning, which allows the model to adjust its hidden embedding dimension at inference time. We show that AdaVid-EgoVLP, trained on video-narration pairs from the large-scale Ego4D dataset, matches the performance of the standard EgoVLP on short video-language benchmarks using only half the compute, and even outperforms EgoVLP when given equal computational resources. We further explore the trade-off between frame count and compute on the challenging Diving48 classification benchmark, showing that AdaVid enables the use of more frames without exceeding computational limits. To handle longer videos, we also propose a lightweight hierarchical network that aggregates short clip features, achieving a strong balance between compute efficiency and accuracy across several long video benchmarks.

Figures

Figures reproduced from arXiv: 2504.12513 by the authors.

Figure 1
Figure 1. A single AdaVid-trained video model facilitates inference [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. AdaVid Framework is designed to train video encoders that facilitate adaptive compute-efficient inference. (a) Key component of AdaVid is the Adaptive Transformer Layer, which is designed to handle input tokens of varying dimension sizes up to D. During each training iteration, each layer processes the input tokens with a randomly selected dimension size, enforcing a coarse-to-fine structure in the model’s weights a… view at source ↗
Figure 3
Figure 3. AdaVid-EgoVLP on two EgoMCQ benchmarks: AdaVid-EgoVLP-dec, trained with decreasing dimensions for deeper layers, performs better than AdaVid-EgoVLP-inc which was trained with increasing dimensions. AdaVid-EgoVLP-dec performs better than baselines while using maximum compute re￾sources. The same model also retains high accuracy when evaluated with low compute evaluation configurations from [PITH_FULL_IMAGE:figures/f… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Results on Diving-48: We evaluate AdaVid using various evaluation configurations from [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 6
Figure 6. Figure 6: AdaVid-Agg on LongVideoRetrieval: We evaluate AdaVid-Agg on text-to-video retrieval from a database of very long videos. We evaluate the dimension configurations mentioned in [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: We show four challenging examples from the EgoMCQ(intra) benchmark, each consisting of a text query and five candidate video [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 8
Figure 8. Figure 8: We show four examples from the EgoSchema VideoQA benchmark, each consisting of a video and a question with 5 candidate [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]
Figure 9
Figure 9. Figure 9: AdaVid-Agg: A long video is divided into S = 16 shorter segments and encoded using the pretrained AdaVid-EgoVLP model. The sequence of segment features is then processed by the lightweight AdaVid-Agg transformer model, which predicts a single feature representation for…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

51 extracted references · 25 canonical work pages

  1. [1]

    Hiervl: Learning hierarchical video-language embeddings

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kris- ten Grauman. Hiervl: Learning hierarchical video-language embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23066–23078, 2023. 2, 3, 5, 6, 7, 11

    work page 2023

  2. [2]

    Frozen in time: A joint video and image encoder for end-to- end retrieval

    Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to- end retrieval. In IEEE International Conference on Computer Vision, 2021. 1, 2, 3, 7

    work page 2021

  3. [3]

    Mem- ory consolidation enables long-context video understanding

    Ivana Balaževi´c, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. Mem- ory consolidation enables long-context video understanding. arXiv preprint arXiv:2402.05861, 2024. 2, 3, 7

    arXiv 2024

  4. [4]

    Long- former: The long-document transformer

    Iz Beltagy, Matthew E Peters, and Arman Cohan. Long- former: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020. 1, 2

    arXiv 2004

  5. [5]

    Is space-time attention all you need for video understanding? In ICML, page 4, 2021

    Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, page 4, 2021. 1, 2, 3, 4, 7, 11

    work page 2021

  6. [6]

    Flexivit: One model for all patch sizes

    Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Min- derer, Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14496–14506, 2023. 2

    work page 2023

  7. [7]

    Once-for-all: Train one network and specialize it for efficient deployment

    Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. arXiv preprint arXiv:1908.09791,

    arXiv 1908

  8. [8]

    Emerg- ing properties in self-supervised vision transformers

    Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerg- ing properties in self-supervised vision transformers. In Pro- ceedings of the IEEE/CVF international conference on com- puter vision, pages 9650–9660, 2021. 3

    work page 2021

Show all 51 references
  1. [9]

    Vision transformer slimming: Multi-dimension searching in continuous optimiza- tion space

    Arnav Chavan, Zhiqiang Shen, Zhuang Liu, Zechun Liu, Kwang-Ting Cheng, and Eric P Xing. Vision transformer slimming: Multi-dimension searching in continuous optimiza- tion space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4931–4941,

  2. [10]

    To- wards the limit of network quantization

    Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. To- wards the limit of network quantization. arXiv preprint arXiv:1612.01543, 2016. 2

    2016 arXiv

  3. [11]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. 2

    2018 arXiv

  4. [12]

    Matformer: Nested transformer for elastic inference

    Fnu Devvrit, Sneha Kudugunta, Aditya Kusupati, Tim Dettmers, Kaifeng Chen, Inderjit S Dhillon, Yulia Tsvetkov, Hannaneh Hajishirzi, Sham M Kakade, Ali Farhadi, et al. Matformer: Nested transformer for elastic inference. In Work- shop on Advancing Neural Network Training: Compu...

    2023

  5. [13]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 3, 7

    2019

  6. [14]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2022

  7. [15]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision ...

    2022

  8. [16]

    Dynamic convnets on tiny devices via nested sparsity

    Matteo Grimaldi, Luca Mocerino, Antonio Cipolletta, and Andrea Calimera. Dynamic convnets on tiny devices via nested sparsity. IEEE Internet of Things Journal, 10(6):5073– 5082, 2022. 2

    2022

  9. [17]

    Transkimmer: Transformer learns to layer-wise skim

    Yue Guan, Zhengyi Li, Jingwen Leng, Zhouhan Lin, and Minyi Guo. Transkimmer: Transformer learns to layer-wise skim. arXiv preprint arXiv:2205.07324, 2022. 1, 2

    2022 arXiv

  10. [18]

    Distill- ing the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distill- ing the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. 2

    2015 arXiv

  11. [19]

    Training compute-optimal large language models

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. 11

    2022 arXiv

  12. [20]

    Dynabert: Dynamic bert with adaptive width and depth

    Lu Hou, Zhiqi Huang, Lifeng Shang, Xin Jiang, Xiao Chen, and Qun Liu. Dynabert: Dynamic bert with adaptive width and depth. Advances in Neural Information Processing Sys- tems, 33:9782–9793, 2020. 2

    2020

  13. [21]

    Long movie clip classification with state-space video models

    Md Mohaiminul Islam and Gedas Bertasius. Long movie clip classification with state-space video models. In European Conference on Computer Vision , pages 87–104. Springer,

  14. [22]

    Video re- cap: Recursive captioning of hour-long videos

    Md Mohaiminul Islam, Ngan Ho, Xitong Yang, Tushar Na- garajan, Lorenzo Torresani, and Gedas Bertasius. Video re- cap: Recursive captioning of hour-long videos. arXiv preprint arXiv:2402.13250, 2024. 6

    2024 arXiv

  15. [23]

    Perceiver io: A general architecture for structured inputs & outputs

    Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Kop- pula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021. 1, 2

    2021 arXiv

  16. [24]

    Matryoshka representation learning

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. Matryoshka representation learning. In Advances in Neural Information Processing Systems, 2022. 2, 5, 8

    2022

  17. [25]

    Block pruning for faster transformers

    François Lagunas, Ella Charlaix, Victor Sanh, and Alexan- der M Rush. Block pruning for faster transformers. arXiv preprint arXiv:2109.04838, 2021. 2

    2021 arXiv

  18. [26]

    Resound: Towards action recognition without representation bias

    Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. InProceedings of the European Conference on Computer Vision (ECCV) , pages 513–528, 2018. 2, 5 9

    2018

  19. [27]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. arXiv preprint arXiv:2206.01670,

    arXiv

  20. [28]

    Egocentric video-language pretraining

    Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wen- zhe Zhao, Weijie Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Sys- tems, 35:7575–7586, 2022. 3

    2022

  21. [29]

    Rethinking the value of network pruning

    Zhuang Liu, Mingjie Sun, Tinghui Zhou, Gao Huang, and Trevor Darrell. Rethinking the value of network pruning. arXiv preprint arXiv:1810.05270, 2018. 2, 3

    2018 arXiv

  22. [30]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 7

    2017 arXiv

  23. [31]

    Univl: A unified video and language pre-training model for multimodal understanding and generation

    Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353, 2020. 3

    2002 arXiv

  24. [32]

    Egoschema: A diagnostic benchmark for very long- form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long- form video language understanding. Advances in Neural Information Processing Systems, 36, 2024. 5, 6, 8

    2024

  25. [33]

    Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips

    Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred mil- lion narrated video clips. In Proceedings of the IEEE/CVF international conference on computer vision , pag...

    2019

  26. [34]

    End-to-end learn- ing of visual representations from uncurated instructional videos

    Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learn- ing of visual representations from uncurated instructional videos. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, page...

    2020

  27. [35]

    A sim- ple recipe for contrastively pre-training video-first encoders beyond 16 frames

    Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joe Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisserman, and Aida Nematzdeh. A sim- ple recipe for contrastively pre-training video-first encoders beyond 16 frames. In Proceedings of the IE...

    2024

  28. [36]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  29. [37]

    Sharcs: Efficient trans- formers through routing with dynamic width sub-networks

    Mohammadreza Salehi, Sachin Mehta, Aditya Kusupati, Ali Farhadi, and Hannaneh Hajishirzi. Sharcs: Efficient trans- formers through routing with dynamic width sub-networks. arXiv preprint arXiv:2310.12126, 2023. 3, 4, 6

    2023 arXiv

  30. [38]

    Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. 1, 2, 4

    1910 arXiv

  31. [39]

    Q- bert: Hessian based ultra low precision quantization of bert

    Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q- bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8815–8821, 2020. 2

    2020

  32. [40]

    Training data-efficient image transformers & distillation through atten- tion

    Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through atten- tion. In International conference on machine learning, pages 10347–10357. PMLR, 2021. 2

    2021

  33. [41]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. 2, 5

    2017

  34. [42]

    Deformable video trans- former

    Jue Wang and Lorenzo Torresani. Deformable video trans- former. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition , pages 14053–14062,

  35. [43]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung- Levy. Videoagent: Long-form video understanding with large language model as agent. arXiv preprint arXiv:2403.10517,

    arXiv

  36. [44]

    Internvideo: General video foundation models via generative and discriminative learning

    Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint arXiv:2212.03191,

    arXiv

  37. [45]

    Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition

    Chao-Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit: Memory-augmented multiscale vision transformer for efficient long-term video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and P...

    2022

  38. [46]

    Videoclip: Contrastive pre-training for zero-shot video-text understanding

    Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. arXiv preprint arXiv:2109.14084, 2021. 1, 2, 3

    2021 arXiv

  39. [47]

    Slimmable neural networks

    Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv:1812.08928, 2018. 2

    2018 arXiv

  40. [48]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024. 7

    2024

  41. [49]

    Poa: Pre-training once for models of all sizes

    Yingying Zhang, Xin Guo, Jiangwei Lao, Lei Yu, Lixiang Ru, Jian Wang, Guo Ye, Huimei He, Jingdong Chen, and Ming Yang. Poa: Pre-training once for models of all sizes. In European Conference on Computer Vision, pages 131–148. Springer, 2024. 3

    2024

  42. [50]

    Mgsampler: An explainable sampling strategy for video ac- tion recognition

    Yuan Zhi, Zhan Tong, Limin Wang, and Gangshan Wu. Mgsampler: An explainable sampling strategy for video ac- tion recognition. In Proceedings of the IEEE/CVF Inter- national conference on Computer Vision, pages 1513–1522,

  43. [2021]

    FLOPs computation A.1

    3 10 AdaVid: Adaptive Video-Language Pretraining Supplementary Material A. FLOPs computation A.1. Multi-Head Self-Attention We follow [ 19] for calculating the total FLOPs of Multi- head self attention as 8ND 2 + 4N 2D, whereN represents the number of tokens andD denotes the t...

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.