{"id":"1dbd2d00-a5b4-43ad-9839-7adddcf07fa6","arxiv_id":"2504.12513","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AdaVid trains video-language encoders whose hidden dimensions can be stripped down at inference time, matching a standard model at half the FLOPs on EgoMCQ.","lead":"AdaVid is a training framework that lets a single video-language model change its internal embedding size at inference time, trading compute for accuracy on the fly. The paper shows the approach matches or beats a standard video encoder on egocentric benchmarks while using half the compute, making it relevant for edge devices and long video understanding.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over EgoVLP are confounded: AdaVid-EgoVLP is initialized from EgoVLP and finetuned for 10 epochs, while the EgoVLP baseline receives no equivalent finetuning, so the 'outperforms at equal compute' claim lacks a controlled comparison.","rationale":"The reader identified the same load-bearing concern: the lack of a controlled finetuned EgoVLP baseline makes the headline comparison uninterpretable. Section 4.3 candidly admits the EgoVLP-initialized, 10-epoch finetuning, so the issue is not hidden, but it is unresolved. The paper's internal compute-accuracy trade-offs across embedding-dimension configurations are plausible and the FLOPs accounting in the supplementary is transparent, which supports the conditional acceptance. However, the central claim that AdaVid 'outperforms EgoVLP when given equal computational resources' depends on isolating the adaptive mechanism from the finetuning recipe. A simple control experiment would settle this: finetune vanilla EgoVLP with identical compute and initialization, without adaptive dimensions. If the control matches AdaVid's EgoMCQ numbers, the paper's contribution reduces to a finetuning recipe rather than a novel adaptive-architecture advantage; if not, the claim holds. The absence of error bars and released code are secondary but also weaken the quantitative claims. I therefore agree with the reader's CONDITIONAL verdict and would not change it; the condition should explicitly include the vanilla-finetuned control.","tokens_in":15543,"tokens_out":7215,"duration_ms":70453,"concrete_test":"Finetune the standard (non-adaptive) EgoVLP using the exact AdaVid recipe: initialize with the same EgoVLP checkpoint, train for 10 epochs with batch size 160 on the same Ego4D narration pairs, using the same learning rate, EgoNCE loss, and all other hyperparameters, but with all embedding dimensions fixed at 768. Evaluate this 'EgoVLP-FT' control on EgoMCQ (inter and intra) and report compute at d-768. If EgoVLP-FT reaches or exceeds AdaVid-EgoVLP's 90.8/59.5, the adaptive procedure contributes nothing beyond finetuning, and the abstract's 'outperforms EgoVLP' claim is unsubstantiated. If EgoVLP-FT remains clearly below AdaVid's numbers, the gain is at least partly attributable to the adaptive width training.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that AdaVid-EgoVLP matches or beats vanilla EgoVLP at equal or half compute. Section 4.3 explicitly states: 'To compensate for our lower batch size compared to EgoVLP (160 vs. 512), and to speed up AdaVid training, we initialize our model with EgoVLP weights and finetune it with adaptive embedding dimensions.' The EgoVLP baseline numbers in Table 2 appear to come from the original EgoVLP model, which was not subject to this additional finetuning. Thus the comparison conflates three variables: (1) 10 extra epochs of training on Ego4D, (2) initialization from a strong EgoVLP checkpoint, and (3) the adaptive width mechanism. The paper attributes the improvement to 'the implicit dropout-like regularization provided by AdaVid training,' but this attribution is unsupported without a control that finetunes vanilla EgoVLP under the same recipe (same epochs, batch size, initialization, hyperparameters, and EgoNCE loss) while keeping dimensions fixed at 768. If such a control matches AdaVid-EgoVLP's 90.8/59.5 on EgoMCQ, the reported advantage is entirely due to finetuning, not the adaptive procedure. This issue is load-bearing because the paper's headline claim about outperforming EgoVLP at equal compute is what distinguishes it from prior adaptive-width work. The Diving-48 comparison with TimeSformer is also confounded by pretraining data (ImageNet-21K vs. Ego4D) and by AdaVid's EgoVLP initialization, but the primary claim is the EgoVLP comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AdaVid, an adaptive transformer framework that lets a single video encoder be evaluated at multiple hidden widths by slicing nested weight matrices, following the Matryoshka Representation Learning idea. The authors instantiate AdaVid-EgoVLP, an adaptive variant of EgoVLP trained on Ego4D video-narration pairs, and AdaVid-Agg, a lightweight aggregator for long videos. They evaluate on EgoMCQ, Diving-48, SummaryMCQ, LongVideoRetrieval, and EgoSchema, and report that AdaVid-EgoVLP matches or outperforms vanilla EgoVLP at half or equal compute, and that adaptive width lets the model process more frames within a fixed FLOP budget. The paper also studies whether embedding dimensions should decrease or increase across layers during training.","tokens_in":15947,"tokens_out":7421,"duration_ms":75316,"significance":"If the claims were established, the contribution would be practically significant: a single video-language encoder that serves multiple compute budgets at inference time, with no distillation or post-hoc pruning, and with a simple nested-weight design. The within-model compute-accuracy curves across d-768, d-576, d-384, and d-192 are a genuine strength and appear to support graceful accuracy degradation with reduced width. The FLOPs formulas in Appendix A are transparent and easy to check. However, the headline comparison to EgoVLP is not yet controlled, because AdaVid-EgoVLP is initialized from EgoVLP weights and additionally finetuned, while the EgoVLP baseline is not. The Diving-48 comparison with TimeSformer is also confounded by different pretraining data. The adaptive-width idea is plausible and the framework is simple, but the central attribution claim needs a control experiment before the reported advantages can be assigned to the method.","major_comments":[{"comment":"The comparison of AdaVid-EgoVLP (90.8/59.5 on EgoMCQ) with vanilla EgoVLP (90.6/57.2) conflates the adaptive-width mechanism with two additional interventions. Section 4.3 states that AdaVid-EgoVLP is initialized from EgoVLP weights and finetuned for 10 epochs with a batch size of 160, whereas the EgoVLP baseline numbers appear to come from the original model without this additional finetuning. To support the abstract's claim that AdaVid \"even outperforms EgoVLP when given equal computational resources,\" the authors must add a control that finetunes vanilla EgoVLP under the identical recipe (same epochs, batch size, initialization, hyperparameters, and EgoNCE loss) while keeping the embedding dimension fixed at 768. If that control reaches 90.8/59.5, the reported advantage is due to extra finetuning rather than to AdaVid; if it does not, the comparison still needs the control to attribute the gain. The statement in Section 3.3 that AdaVid-trained models \"often outperform their vanilla counterparts despite identical training setups\" is unsupported because no identical-training-setup vanilla model is evaluated.","section":"Section 4.3, Table 2"},{"comment":"The Diving-48 comparison is confounded. The text says TimeSformer shares the same architecture, \"ensuring a fair comparison,\" but the table caption states that the baselines are pretrained on ImageNet-21K while the AdaVid models are pretrained on Ego4D; additionally, AdaVid-EgoVLP-FT starts from EgoVLP weights and is finetuned on Diving-48. Therefore the frame-count-versus-compute claim, for example 128 frames at 285e10 FLOPs reaching 82.2% versus TimeSformer-L at 714e10 FLOPs reaching 81.0%, does not isolate the adaptive-width contribution. At minimum, these results should be reworded as cross-paper comparisons rather than controlled experiments, and an apples-to-apples control with the same pretraining data and finetuning recipe, using a non-adaptive baseline, should be run if the \"more frames within limited compute\" claim is to be attributed to the method.","section":"Section 4.3, Table 3"},{"comment":"The dimension sampling strategy is underspecified, which blocks reproduction of the dec-versus-inc finding. Section 3.3 says a dimension is \"chosen randomly or based on some strategy\" from {D, 3D/4, D/2, D/4}, but the paper never specifies the sampling distribution, whether each layer is sampled independently per iteration, how the decreasing or increasing constraint is enforced within a step, or whether the schedule changes over the 10 epochs. The claims that \"gradually decreasing embedding dimension sizes across layers yields better performance\" and that the opposite strategy \"bottlenecks the information at early layers\" depend on this exact procedure. Please provide the full sampling algorithm, including the random seed policy, or release code; without this, the central training recipe is not precisely defined.","section":"Section 3.3 / Section 4.3"}],"minor_comments":[{"comment":"There is a typo in \"compute-contrained\" which should be \"compute-constrained.\"","section":"Section 2"},{"comment":"The TimeSformer citations are inconsistent: the table cites [2] (Frozen in Time) where [5] (TimeSformer) is presumably intended.","section":"Table 3"},{"comment":"The source of the EgoVLP baseline numbers should be stated explicitly, namely whether they are taken from the original paper or re-evaluated with the same evaluation code, and confidence intervals or repeated-seed variance for EgoMCQ would help interpret the 0.2-point inter and 2.3-point intra gaps.","section":"Section 4.3, Table 2"},{"comment":"The LongVideoRetrieval benchmark is newly curated in this paper, but no dataset URL or release instructions are given; please include this information for reproducibility.","section":"Section 4.2"},{"comment":"There is a typo in the table caption: \"signficantly\" should be \"significantly.\"","section":"Section 4.4, Table 5"}],"recommendation":"major_revision","confidential_remarks":"The main confound identified in Section 4.3 is real and is fixable by adding a control experiment that finetunes vanilla EgoVLP under the identical recipe. If the authors can run that control and it confirms the adaptive-width advantage, the paper would be a solid contribution. The within-model compute-accuracy analysis is already useful even if the cross-model comparison is reworded. I would not reject on the current evidence, but the headline claims should not be published without the control."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper has a real contribution: it extends width-adaptive transformers to every layer of a video-language model, including self-attention via head-count reduction, and shows that a decreasing-width training schedule works better than an increasing one. The internal accuracy-vs-FLOP curves from one checkpoint (d-768 down to d-192) are coherent, and the appendix FLOPs math is transparent. That part holds up.\n\nThe soft spot is exactly what the stress-test note says. Section 4.3 admits AdaVid-EgoVLP is initialized from EgoVLP weights and finetuned for 10 epochs at batch size 160, while the EgoVLP baseline numbers come from the original model with no such finetuning. So the headline claim 'outperforms EgoVLP at equal compute' conflates the adaptive mechanism with extra training and a strong starting point. The paper's explanation — implicit dropout-like regularization — is plausible but unsupported without a control that finetunes vanilla EgoVLP under the same recipe. That control is the single most important missing experiment.\n\nThe Diving-48 comparison has a similar issue: AdaVid-EgoVLP-FT is pretrained on Ego4D and initialized from EgoVLP, while the TimeSformer baselines are pretrained on ImageNet-21K. Not a fair comparison, and the paper's claim that TimeSformer 'shares the same architecture' doesn't erase the pretraining gap.\n\nMinor issues: no error bars, no code release, and LongVideoRetrieval is self-curated without a human check that captions are distinctive enough. None of these are fatal.\n\nOverall, this is a genuine engineering effort with a clear mechanism and mostly honest reporting — the confound is disclosed, not hidden. But as written, the central comparative claim is not established. The paper deserves a serious referee. I'd ask for a vanilla-finetuned-EgoVLP control and code release; those two changes would turn a conditional result into a solid one.","headline":"Solid adaptive-width video encoder, but the headline EgoVLP comparison needs a controlled finetuned baseline before it convinces.","tokens_in":16466,"tokens_out":2394,"would_cite":true,"duration_ms":24406,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AdaVid makes a video encoder's hidden width adjustable at inference, so one pretrained model covers many compute budgets and beats its fixed-width baseline at matched cost.","keywords":["adaptive computation","video-language pretraining","transformer efficiency","Matryoshka representation learning","egocentric video","long video understanding","embedding dimension","edge deployment"],"falsifier":"Train vanilla EgoVLP with the same recipe used for AdaVid-EgoVLP, meaning the same epoch count, batch size, and EgoVLP initialization but with fixed full width, and compare on EgoMCQ at equal FLOPs. If vanilla finetuning closes the gap, adaptive-width training is not the cause of the reported advantage; a further check is pretraining AdaVid-EgoVLP from scratch and measuring whether it still matches EgoVLP at half compute.","tokens_in":15353,"feed_emoji":"🎬","tokens_out":6501,"duration_ms":62255,"temperature":0.7,"pith_summary":"The paper proposes that a video-language encoder can be trained so that the same network runs at widely different compute costs by slicing each transformer layer's hidden dimension at inference time. In this AdaVid framework, each layer is trained with randomly chosen widths, so a single model effectively contains many smaller models and needs no postprocessing to switch between them. On egocentric video-text pretraining, AdaVid-EgoVLP matches the standard EgoVLP at roughly half the compute and outperforms it when given equal compute. The same adaptive model can also spend its compute savings on more input frames, which helps on long-video tasks like Diving-48. A lightweight aggregator, AdaVid-Agg, extends the idea to long videos and matches a stronger hierarchical baseline at lower cost.","feed_headline":"Adaptive video encoder matches full model at half the compute","feed_subtitle":"A single AdaVid model runs fast or accurate on demand, and at equal cost beats the standard EgoVLP baseline.","key_machinery":"The adaptive transformer layer is the mechanism that carries the argument: each linear projection $y = Wx + b$ is restricted to the upper-left $d\\times d$ submatrix $W[1:d,1:d]$ and the first $d$ bias entries, layer normalization uses the first $d$ scale and shift values, and multi-head attention uses $d/H$ heads instead of shrinking each head. Randomly sampling widths during pretraining forces a coarse-to-fine structure in the weights and activations, so the trained model can later be evaluated at any allowed width, and the paper finds this acts as a mild regularizer.","core_discovery":"The central claim is that the computational cost of a video transformer can be made adjustable without retraining or postprocessing by making the embedding dimension of every layer adaptive. During training the layer width is sampled randomly from the set $\\{D, 3D/4, D/2, D/4\\}$; at inference the model drops trailing dimensions and uses the corresponding leading submatrices of its weights, with attention using fewer heads rather than narrower heads. AdaVid-EgoVLP, pretrained on Ego4D with the EgoNCE objective, scores 90.8/59.5 on EgoMCQ inter/intra versus 90.6/57.2 for vanilla EgoVLP at full width, and retains most of that accuracy at half compute. On Diving-48, AdaVid-EgoVLP-FT at 128 frames and 285e10 FLOPs reaches 82.2%, ahead of TimeSformer-L at 96 frames and 714e10 FLOPs. The paper also claims that a decreasing-width schedule, with wide early layers and narrow deep layers, works better than the reverse.","pith_inferences":["Because the slicing mechanism is stated purely in terms of weight submatrices, it should transfer to other transformer-based encoders, including image-language models; the paper only demonstrates it on video.","An untested but natural deployment pattern is two-stage retrieval: rank candidates with a narrow width and rerank the top few at full width, an approach the paper mentions as future work.","A from-scratch training run would separate the regularization effect from the finetuning effect; if the equal-compute advantage survives, adaptive-width pretraining could become a standard recipe for contrastive video models rather than a special-purpose efficiency trick."],"forward_implications":["At full width, AdaVid training acts as a regularizer: AdaVid-EgoVLP beats vanilla EgoVLP on EgoMCQ even though it is the same architecture trained with the same supervision.","A fixed compute budget can be spent on more frames: AdaVid-EgoVLP-FT reaches 82.2% on Diving-48 with 128 frames and 285e10 FLOPs, where TimeSformer-L needs 714e10 FLOPs for 81.0%.","Decreasing width across layers is the better design: decreasing configurations outperform increasing ones at matched FLOPs on EgoMCQ, consistent with deeper layers storing only high-level concepts.","Long-video understanding inherits the same flexibility: AdaVid-Agg matches HierVL on SummaryMCQ and EgoSchema at full width and retains most of its accuracy at 0.25x compute.","A single AdaVid model gives a smooth compute-accuracy operating curve, from about 0.06x to 1.0x of the reference model's FLOPs, so deployment can pick the width that fits the available device budget."],"supporting_citations":[{"why":"EgoVLP is the architecture and supervision AdaVid adapts, and the vanilla baseline it must match or beat; it supplies the EgoNCE loss and the EgoMCQ benchmark.","marker":"[27]"},{"why":"Matryoshka Representation Learning supplies the adaptive-dimension principle: leading-dimension subvectors of representations are trained to stay useful.","marker":"[24]"},{"why":"TimeSformer space-time attention is the backbone of the adaptive layers and provides the Diving-48 comparison baseline TimeSformer-L.","marker":"[5]"},{"why":"HierVL is the hierarchical long-video baseline AdaVid-Agg is compared against, and it supplies the summary-supervision setup and SummaryMCQ benchmark.","marker":"[1]"},{"why":"Ego4D provides the video-narration and summary pairs used for pretraining AdaVid-EgoVLP and AdaVid-Agg.","marker":"[14]"},{"why":"SHARCS is the prior adaptive-width transformer whose design choices, fewer heads instead of narrower heads and deep-layer width reduction, AdaVid builds on.","marker":"[37]"},{"why":"Diving-48 supplies the classification benchmark used to show the frame-count versus compute trade-off.","marker":"[26]"},{"why":"EgoSchema supplies the long-form video QA benchmark used to show low-compute robustness.","marker":"[32]"}],"fun_headline_variants":["Adaptive video model matches full accuracy at half compute","One video model, any compute budget: accuracy scales with cost","Video transformer adjusts its own size for speed or accuracy","One encoder, many speeds: same accuracy at half cost","Video AI that shrinks itself: full accuracy, half the compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison to vanilla EgoVLP assumes that AdaVid-EgoVLP's gains come from adaptive-width training rather than from its extra 10 epochs of finetuning and its initialization from already-trained EgoVLP weights; vanilla EgoVLP receives no equivalent finetuning.","fun_headline_variants_meta":{"raw":{"variants":["Adaptive video model matches full accuracy at half compute","One video model, any compute budget: accuracy scales with cost","Video transformer adjusts its own size for speed or accuracy","One encoder, many speeds: same accuracy at half cost","Video AI that shrinks itself: full accuracy, half the compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000854,"raw_usage":{"total_tokens":3746,"prompt_tokens":1015,"completion_tokens":2731,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":2649}},"tokens_in":631,"tokens_out":2731,"duration_ms":20670,"temperature":1.0,"reasoning_tokens":2649,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:30:36.785898+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train vanilla EgoVLP with the same recipe used for AdaVid-EgoVLP, meaning the same epoch count, batch size, and EgoVLP initialization but with fixed full width, and compare on EgoMCQ at equal FLOPs. If vanilla finetuning closes the gap, adaptive-width training is not the cause of the reported advantage; a further check is pretraining AdaVid-EgoVLP from scratch and measuring whether it still matches EgoVLP at half compute.","supporting_citations":[{"cited_title":"Matryoshka representation learning","cited_arxiv_id":null,"evidence_quote":"Matryoshka Representation Learning supplies the adaptive-dimension principle: leading-dimension subvectors of representations are trained to stay useful."},{"cited_title":"Is space-time attention all you need for video understanding? In ICML, page 4, 2021","cited_arxiv_id":null,"evidence_quote":"TimeSformer space-time attention is the backbone of the adaptive layers and provides the Diving-48 comparison baseline TimeSformer-L."},{"cited_title":"Hiervl: Learning hierarchical video-language embeddings","cited_arxiv_id":null,"evidence_quote":"HierVL is the hierarchical long-video baseline AdaVid-Agg is compared against, and it supplies the summary-supervision setup and SummaryMCQ benchmark."},{"cited_title":"Ego4d: Around the world in 3,000 hours of egocentric video","cited_arxiv_id":null,"evidence_quote":"Ego4D provides the video-narration and summary pairs used for pretraining AdaVid-EgoVLP and AdaVid-Agg."},{"cited_title":"SHARCS: Efficient Transformers through Routing with Dynamic Width Sub-networks","cited_arxiv_id":"2310.12126","evidence_quote":"SHARCS is the prior adaptive-width transformer whose design choices, fewer heads instead of narrower heads and deep-layer width reduction, AdaVid builds on."},{"cited_title":"Resound: Towards action recognition without representation bias","cited_arxiv_id":null,"evidence_quote":"Diving-48 supplies the classification benchmark used to show the frame-count versus compute trade-off."},{"cited_title":"Egoschema: A diagnostic benchmark for very long- form video language understanding","cited_arxiv_id":null,"evidence_quote":"EgoSchema supplies the long-form video QA benchmark used to show low-compute robustness."}],"review_version":1}