Pith. sign in

REVIEW 1 major objections 7 minor 23 references

Video diffusion models are already competent visual encoders

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-10 00:06 UTC pith:2N4SP2IZ

load-bearing objection Frozen video diffusion models produce genuinely useful representations for classification and geometry, but the captioning results undercut the 'wide spectrum' framing. the 1 major comments →

arxiv 2607.06856 v1 pith:2N4SP2IZ submitted 2026-07-07 cs.CV cs.LG

Gen4U: Unifying Video Generation and Understanding via Diffusion

Michael King , Aravindh Mahendran , Matthew Koichi Grimes , Fedor Kitashov , Adham Elarabawy , Pedro Velez , Maks Ovsjanikov , Viorica P\u{a}tr\u{a}ucean This is my paper
classification cs.CV cs.LG
keywords video diffusion modelsrepresentation learningvideo understandingfrozen featuresmutual k-NN alignmentgenerative models for perceptionlatent space probingvideo classification
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that state-of-the-art video diffusion models, trained purely to generate video, implicitly learn rich visual representations spanning both high-level semantics and low-level geometry. By probing intermediate activations across network depth and noise levels, the authors identify a structured latent space where moderate noise yields linearly separable global semantics while lower noise preserves fine-grained but spatially scattered details. They introduce Gen4U, which extracts representations from a single optimal layer and noise level in a single forward pass, pairing them with lightweight task-specific decoders. Without modifying the frozen generative weights, the approach achieves competitive performance on video classification, depth estimation, camera pose estimation, and captioning, suggesting that a single diffusion backbone can serve simultaneously as a generator and a general-purpose perception encoder.

Core claim

The central finding is that scaling video diffusion models to high generative fidelity causes general-purpose visual representations to emerge implicitly in intermediate activations, without any explicit contrastive or reconstruction-based understanding objective. The authors map this emergence along two axes, network depth and noise level, and identify a consistent sweet spot (roughly 75-80% depth, 30-60% noise) where features align with both text encoders and discriminative visual encoders. This alignment is strong enough that a single forward pass through the frozen diffusion backbone, paired with a lightweight decoder, matches or approaches specialized encoders across semantic and non-ge

What carries the argument

The mechanism is the identification of an optimal extraction point within the diffusion backbone's two-dimensional latent grid (depth x noise level), from which a single forward pass yields features competitive with discriminative encoders. The mutual k-NN alignment metric serves as the probe that locates this point by measuring neighborhood overlap between diffusion activations and reference encoders without any learned mapping.

Load-bearing premise

The claim that a single extraction point generalizes across all tasks rests on the fact that different probes and tasks have different optima within the depth-noise grid, and the captioning results show substantial gaps to specialized baselines, meaning the 'wide spectrum' of competitiveness is narrower for language-grounded tasks than for classification or geometry.

What would settle it

If a discriminative video encoder trained with equivalent compute and data scale were evaluated on the same task suite, and it matched or exceeded the diffusion features on all tasks including generation-adjacent ones, then the argument that generation is a sufficient or efficient path to general representations would weaken. More directly, if future, more capable diffusion models showed declining rather than improving representation alignment, the scaling claim would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • A single frozen diffusion backbone could replace separate generation and understanding models, reducing training and deployment costs for video AI systems.
  • The finding that semantic structure emerges at a specific noise level (around 60%) provides a concrete target for future probes of generative model internals.
  • The bimodal depth pattern observed in the proprietary model suggests that large transformers may route information through a contraction-expansion cycle, a hypothesis testable across architectures.
  • If stronger generative models yield richer representations, as the authors observe, then continued scaling of video generation may produce increasingly capable understanding models for free.
  • The noise-augmentation strategy for captioning, where multiple noise levels serve as data augmentation, offers a practical recipe for extracting more from frozen diffusion features on limited training data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the correlation between generative capability and representation quality holds, then the quality of a diffusion model's internal features could serve as a proxy metric for generation fidelity, potentially useful for model selection before expensive generation evaluation.
  • The task-dependent shift between linear and attention probe optima (60% vs 30% noise) hints that the diffusion process encodes a spectrum from global gist to local detail, and a multi-extraction architecture that reads from multiple noise levels simultaneously might outperform any single-point extraction.
  • The gap between diffusion features and discriminative baselines on captioning suggests that language-grounded semantics remain partially external to the generative objective; a model trained with both generation and explicit language alignment might close this gap without sacrificing generation quality.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

1 major / 7 minor

Summary. The paper investigates whether frozen, large-scale video diffusion models (Veo3 and Wan 2.2) can serve as general-purpose video encoders. Using mutual k-NN alignment metrics and linear/attention probes, the authors map the structure of the diffusion latent space across network depth and noise levels, identifying an optimal extraction point (depth ~75-80%, noise ~30-60%). They then evaluate frozen representations on video classification (SSv2), depth estimation (ScanNet), camera pose estimation, and image/video captioning (COCO, Vatex, SSv2). The SSv2 classification (71.3% Top-1) and depth estimation (AbsRel 0.075) results are genuinely strong. The central claim that modern diffusion models have overcome the semantic limitation identified in prior work (Vélez et al., 2025) is a meaningful contribution. However, the breadth of the 'highly competitive across a wide spectrum of tasks' claim is not uniformly supported by the data, particularly for captioning tasks where gaps to discriminative baselines are large.

Significance. The paper addresses a timely and important question: whether the generative paradigm, specifically video diffusion, can produce representations competitive with discriminative encoders across both semantic and geometric tasks. The latent space analysis (§3) is systematic and the identification of a single-pass extraction point is practically valuable. The SSv2 classification result (Table 1) and depth estimation result (AbsRel 0.075, a 10.7% relative improvement over the frozen-feature baseline of 4DS) are concrete achievements. The finding that stronger generative models yield richer representations (Veo3 vs. Wan 2.2 alignment scores) is a falsifiable and useful observation. The work is likely to stimulate further research into diffusion representations.

major comments (1)
  1. The abstract claims frozen diffusion models are 'highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning).' However, Table 2 shows this is not the case for captioning: on COCO, Gen4U achieves CIDEr 54.9 vs. 204.5 for SigLIP-so400m/14 (a 3.7× gap). Even with noise augmentation and doubled resolution, the best COCO CIDEr is 102.0 — still half the SigLIP baseline. On Vatex, the gap is 44.8 vs. 66.0 (or 56.7 with augmentation). The paper itself acknowledges this internally (§4.1: 'video diffusion representations excel on SSv2 but struggle on COCO and Vatex'), but the abstract and conclusion do not reflect this limitation. The semantic competence is convincingly demonstrated primarily for SSv2 classification (a discrete label prediction task), a
minor comments (7)
  1. Table 1: The model size for Gen4U is listed as '-' while all baselines report parameter counts. This makes it difficult to assess whether the comparison is parameter-controlled. Please provide the backbone size or at least note it explicitly.
  2. §3.4, 'Feature routing across network depth': The bimodal pattern observed for Veo3 is described qualitatively and hypothesized to mirror the geometry of large transformer representations (Valeriani et al., 2023). The claim would be strengthened by a quantitative measure (e.g., rank, intrinsic dimension, or entropy) across depth.
  3. Figure 3: The y-axis label 'Alignment Score' and the scale (0.00–0.10) are consistent across subplots, but the caption does not clarify whether the same N=1024 videos and k=10 are used for all panels. A clarifying note would help.
  4. §4.1, Image and video captioning: The paper states that the setup 'does not involve tuning the adapter and LLM decoder on large vision-language data, so it is not directly comparable to SoTA methods like BLIP-2 and PaLI Gemma.' This is fair, but the SigLIP baselines used for comparison are described as 'highly optimised for alignment with text.' Please clarify whether the SigLIP baselines also benefit from large-scale VLM pre-training, to ensure the comparison is fair.
  5. Appendix B, Table 3: The SSv2 subset experiment (17.04% baseline, 24.9% with cross-attention adapter) uses only 100 samples per class. The results are described as 'preliminary.' Please label this more clearly as exploratory and note the limited training set size in the main text reference (§3.4).
  6. §3.1: The text mentions using 'a generic text embedding: e.g. A video of a scene' as conditioning. It would be useful to report results with empty string conditioning quantitatively, at least for one task, to support the claim of 'no significant difference.'
  7. The paper uses 'state-of-the-art' in several places (e.g., §4.1 for SSv2, §4.2 for ScanNet depth). Given that the comparisons are under specific frozen-feature protocols, 'state-of-the-art under this evaluation protocol' or 'best among frozen-feature baselines' would be more precise.

Circularity Check

0 steps flagged

No circularity found: the derivation chain is self-contained against external benchmarks with no fitted-input-as-prediction or self-citation load-bearing steps.

full rationale

The paper's central claim — that frozen video diffusion models serve as competitive video encoders — is established through independent, externally falsifiable evaluations on standard benchmarks (SSv2, ScanNet, COCO, Vatex) using standard metrics (Top-1 accuracy, AbsRel, EPE, CIDEr, BLEU@4). The mutual k-NN alignment metric (Section 3.2, Appendix A) is zero-shot: it constructs nearest-neighbor graphs independently in each embedding space and measures overlap, with no learned mapping between spaces. The 'sweet spot' extraction point (depth ~75-80%, noise ~30-60%) is identified via systematic sweeps (Figures 3-6), not fitted to downstream targets. The lightweight decoders (linear probe, attention probe, DPT head, Gemma-2B adapter) are trained on task training splits and evaluated on held-out test splits following protocols from prior work (Carreira et al. 2025). The paper does cite Vélez et al. 2025 (shared co-author Pedro Velez), but that citation serves as a contrasting baseline — it found diffusion models LACK semantic capabilities — and the present paper argues against that conclusion using independent evidence. The captioning results on COCO and Vatex show gaps to SigLIP baselines, but this is a matter of claim strength (correctness risk), not circularity: the evaluation protocol is standard, the baselines are independently trained models, and the metrics are externally defined. No step in the derivation chain reduces to its inputs by construction.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

No new entities are postulated. The paper works entirely with existing models, metrics, and benchmarks.

free parameters (5)
  • Optimal extraction depth (~75-80%) = 75-80% of network depth
    Selected by sweeping over 12 equally-spaced blocks and reporting the best-performing configuration on probing metrics and downstream tasks.
  • Optimal noise level (30-60%) = 30-60% noise level
    Selected by sweeping over 4 noise levels (10%, 30%, 60%, 90%) and identifying the peak in alignment/probe accuracy.
  • k=10 for mutual k-NN = 10
    Chosen following prior work (Huh et al., 2024); not fitted to the present results but inherited.
  • N=1024 videos for alignment = 1024
    Dataset size for alignment computation; chosen for computational feasibility.
  • Noise augmentation levels for captioning = 10%, 30%, 60%
    Selected on validation split as optimal combination for data augmentation.
axioms (4)
  • domain assumption Mutual k-NN alignment correlates with downstream task performance.
    Invoked in §3.2 to justify using alignment scores as a proxy for representation quality. Supported by Zhu et al., 2026 but treated as given.
  • ad hoc to paper A single forward pass at a fixed noise level captures sufficient information for understanding tasks.
    The entire Gen4U framework rests on this. The paper shows it works empirically but does not derive why a single noise level suffices when diffusion is inherently a multi-step process.
  • domain assumption Text conditioning with a generic prompt ('A video of a scene') does not leak task-specific information.
    Stated in §3.1. The authors verify that empty string conditioning gives similar results, but this is only briefly mentioned.
  • domain assumption Feature alignment scales with generative capability.
    Invoked in §4 to justify focusing downstream evaluation on Veo3 rather than Wan 2.2. Based on the observation that Veo3 shows higher alignment than Wan 2.2.

pith-pipeline@v1.1.0-glm · 21314 in / 3981 out tokens · 156418 ms · 2026-07-10T00:06:14.124521+00:00 · methodology

0 comments
read the original abstract

Prior work suggests that diffusion representations capture low-level geometry but struggle with high-level semantics. We demonstrate that state-of-the-art video diffusion models overcome this limitation. By systematically probing their intermediate activations using recent mutual-kNN alignment metrics, we reveal a highly structured latent space where visual representations evolve across both network depth and noise levels. We show that while moderate noise levels yield linearly separable global semantics, fine-grained details persist at lower noise levels but become spatially scattered, requiring attention mechanisms to decode. Building on these insights, we introduce Gen4U (Generation for Understanding), a framework that repurposes these generative representations with a single forward pass. Our experiments establish that frozen, large-scale video diffusion models function as highly competitive video encoders across a wide spectrum of tasks, spanning semantic and non-semantic objectives (video classification, depth estimation, camera pose estimation, image and video captioning). Bypassing fine-tuning, Gen4U unifies the generation and understanding paradigms, achieving strong perception performance while fully preserving the model's ability to generate high-quality video.

Figures

Figures reproduced from arXiv: 2607.06856 by Adham Elarabawy, Aravindh Mahendran, Fedor Kitashov, Maks Ovsjanikov, Matthew Koichi Grimes, Michael King, Pedro Velez, Viorica P\u{a}tr\u{a}ucean.

Figure 1
Figure 1. Figure 1: Video generative model repurposed as video encoder. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: PCA visualisation of activations extracted from Veo3 at different depths and noise levels. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mutual-kNN zero-shot video-text alignment for Veo3 (left) and Wan 2.2 (right) against [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Alignment between diffusion and discriminative representations using mutual k-NN metric. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Linear (left) and attention (right) probing of Veo3 activations with SSv2 video classification. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance on geometry tasks tasks across blocks and noise levels confirm the sweet [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Captions generated using Gen4U with a Gemma2-2B decoder on the COCO dataset. captioning. Our captioning setup consists of the following two steps: (1.) A cross-attention adapter, inspired by the decoder design of [Saj￾jadi et al., 2022], with three cross-attention blocks, projects the visual representation into 32 learnable query tokens; and (2.) we prepend these 32 tokens to the text input before the [BOS… view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative video reconstruction results on two example videos from the Something [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Evaluating RGB readout heads attached at varying network depths and noise levels. Left: [PITH_FULL_IMAGE:figures/full_fig_p018_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

23 extracted references · 23 canonical work pages · 12 internal anchors

  1. [1]

    V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

    URL https://arxiv.org/abs/ 2506.09985. Adrien Bardes, Quentin Garrido, Jean Ponce, Xinlei Chen, Michael Rabbat, Yann LeCun, Mido Assran, and Nicolas Ballas. Revisiting feature prediction for learning visual representations from video.TMLR,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    PaliGemma: A versatile 3B VLM for transfer

    ISSN 2835-8856. URL https://openreview.net/forum?id= QaCCuDfBk2. Featured Certification. Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, Thomas Unterthiner, et al. PaliGemma: A versatile 3B VLM for transfer.arXiv preprint arXiv:24...

    work page internal anchor Pith review Pith/arXiv arXiv

  3. [3]

    Emerging

    doi: 10.1109/ICCV48922.2021.00951. 10 João Carreira, Dilara Gokay, Michael King, Chuhan Zhang, Ignacio Rocco, Aravindh Mahendran, Thomas Albert Keck, et al. Scaling 4d representations,

    work page doi:10.1109/iccv48922.2021.00951 2021

  4. [4]

    Scaling 4D Representations

    URL https://arxiv.org/abs/ 2412.15212. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. InICML,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Microsoft COCO Captions: Data Collection and Evaluation Server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325,

    work page internal anchor Pith review Pith/arXiv arXiv

  6. [6]

    Whatever next? Predictive brains, situated agents, and the future of cognitive science,

    doi: 10.1017/S0140525X12000477. Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. InCVPR,

    work page doi:10.1017/s0140525x12000477

  7. [7]

    Accessed: 2026-04-17

    URL https: //sander.ai/2024/09/02/spectral-autoregression.html. Accessed: 2026-04-17. David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. InNeurIPS, pages 2366–2374,

    work page 2024

  8. [8]

    Gemma 2: Improving Open Language Models at a Practical Size

    URL https://arxiv.org/abs/2408.00118. Google DeepMind. Veo 3 technical report. Technical report, Google,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    URLhttps://doi.org/10.1038/s41586-025-08744-2

    doi: 10.1038/s41586-025-08744-2. URLhttps://doi.org/10.1038/s41586-025-08744-2. Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross B. Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 15979–15988,

    work page doi:10.1038/s41586-025-08744-2

  10. [10]

    Adam: A Method for Stochastic Optimization

    11 Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    V-JEPA 2.1: Unlocking Dense Features in Video Self-Supervised Learning

    URLhttps://arxiv.org/abs/2603.14482. Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V . V o, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, et al. DINOv2: Learning robust visual features without supervision.TMLR,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    A simple recipe for contrastively pre-training video-first encoders beyond 16 frames.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14386–14397,

    Pinelopi Papalampidi, Skanda Koppula, Shreya Pathak, Justin Chiu, Joseph Heyward, Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisserman, and Aida Nematzdeh. A simple recipe for contrastively pre-training video-first encoders beyond 16 frames.2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14386–14397,

    work page 2024

  13. [13]

    Emerging

    doi: 10.1109/ICCV48922.2021.00129. URL https://doi. ieeecomputersociety.org/10.1109/ICCV48922.2021.00129. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. InCVPR, pages 10684–10695, June

    work page doi:10.1109/iccv48922.2021.00129 2021

  14. [14]

    PaliGemma 2: A Family of Versatile VLMs for Transfer

    12 Andreas Steiner, André Susano Pinto, Michael Tschannen, Daniel Keysers, Xiao Wang, Yonatan Bitton, Alexey Gritsenko, Matthias Minderer, Anthony Sherbondy, Shangbang Long, Siyang Qin, Reeve Ingle, Emanuele Bugliarello, Sahar Kazemzadeh, Thomas Mesnard, Ibrahim Alabdulmohsin, Lucas Beyer, and Xiaohua Zhai. PaliGemma 2: A Family of Versatile VLMs for Tran...

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities

    URLhttps://arxiv.org/abs/2507.06261. Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training.NeurIPS,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features

    Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, Olivier Hénaff, Jeremiah Harmsen, Andreas Steiner, and Xiaohua Zhai. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features.a...

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Wan: Open and Advanced Large-Scale Video Generative Models

    Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.20314,

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    URLhttps://doi.org/10.1007/978-3-031-73013-9_23

    1007/978-3-031-73013-9_23. URLhttps://doi.org/10.1007/978-3-031-73013-9_23. Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, Shixiang Shane Gu, Nick Matarese, Kevin Swersky, Been Kim, Priyank Jaini, and Robert Geirhos. Video models are zero-shot learners and reasoners,

    work page doi:10.1007/978-3-031-73013-9_23

  19. [19]

    Video models are zero-shot learners and reasoners

    URLhttps://arxiv.org/abs/2509.20328. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. InICCV,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    14 A Mutualk-NN alignment metric We describe the Mutual k-Nearest Neighbours (MkNN) alignment metric used throughout the paper, following [Huh et al., 2024, Zhu et al., 2026]

    URL https: //openreview.net/forum?id=gE17TwVMNh. 14 A Mutualk-NN alignment metric We describe the Mutual k-Nearest Neighbours (MkNN) alignment metric used throughout the paper, following [Huh et al., 2024, Zhu et al., 2026]. Notation.Consider a dataset of N videos {vi}N i=1, each paired with some text ci. We compare the representations of two encoders who...

    work page 2024

  21. [21]

    Best Single block

    Linear adapters consistently outperform non-linear ones in terms of zero-shot generalization across datasets and tasks. For instance, a linear adapter pre-trained purely for text alignment on V ATEX 16 Adapter Training data Training objective Accuracy Best single block (baseline) – – 17.04 Linear V ATEX Text alignment 21.4 Linear SSv2 Text alignment 21.9 ...

    work page 2017

  22. [22]

    The linear decoder presents frames that are sharper, but temporally misaligned with the ground truth, as compared to the attention head

    Results.Figure 8 presents frames from the original video and their previews decoded from the outputs at depth 75%, evaluated at noise level 60%. The linear decoder presents frames that are sharper, but temporally misaligned with the ground truth, as compared to the attention head. 18 Figure 9 shows MSE reconstruction metrics for attaching the preview head...

    work page 2024

  23. [23]

    It plateaus after 10k steps so that the LLM is still updated very slowly to avoid catastrophic forgetting

    The Gemma2 weights are initially frozen and gradually thawed with a linear schedule going from 0 to 0.01 over 10k iterations. It plateaus after 10k steps so that the LLM is still updated very slowly to avoid catastrophic forgetting. We use the ADAM optimizer [Kingma and Ba, 2014] with batch size of64for all experiments in this subsection. Models are train...

    work page 2014