Unifying Specialized Visual Encoders for Video Language Models

Honglu Zhou; Jihoon Chung; Juan Carlos Niebles; Max Gonzalez Saez-Diez; Olga Russakovsky; Tyler Zhu

REVIEW 4 major objections 5 minor 2 cited by

Unifying Specialized Visual Encoders for Video Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read MERV, a video-language model that fuses four frozen visual encoders, outperforms single-encoder models on standard video QA benchmarks.

desk verdict A solid, well-ablated empirical recipe for multi-encoder video LLMs whose exact advantage margins need error bars before the headline numbers are trusted. read the letter →

arxiv 2501.01426 v2 pith:PZBKTLET submitted 2025-01-02 cs.CV cs.CLcs.LG

Jihoon Chung , Tyler Zhu , Max Gonzalez Saez-Diez , Juan Carlos Niebles , Honglu Zhou , Olga Russakovsky This is my paper · ORCID

classification cs.CVcs.CLcs.LG

keywords MERVvideolanguagemodelsmulti-encoderfusionspatio-temporalalignmentcross-attentionfeaturemixingfrozenvisualencodersquestionansweringSomething-Somethingv2

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that video-language models unnecessarily limit themselves by letting a single vision encoder supply all visual information, and that combining several specialized encoders into one representation is a cheap way to broaden what the language model can see. The proposed model, MERV (Multi-Encoder Representation of Videos), freezes four encoders — DINOv2 for spatial detail, ViViT for temporal structure, SigLIP for image–text associations, and LanguageBind for video–text semantics — and aligns, projects, then cross-attention-mixes their features before feeding them to a LLaMA-2 7B backbone. On standard video QA benchmarks, MERV outperforms the single-encoder Video-LLaVA model it was adapted from, with gains up to 4.62 accuracy points on TVQA, and the fully trained version reaches 48.4% zero-shot accuracy on the Perception Test, ahead of SeViLA's 46.2%. The added parameters are minimal, and parallel visual processing keeps the per-step training time close to that of a single-encoder model. The paper's intended lesson is that scaling video understanding should include widening the visual encoder set, not just enlarging the language model.

What carries the argument

The load-bearing mechanism is the spatio-temporal pre-fusion projection followed by a cross-attention mixer. A 2D adaptive average pool resizes every encoder's spatial feature map to one common grid, and a per-encoder linear layer maps all feature dimensions to the LLM's dimension; the temporal dimension is matched by choosing input frame counts so each encoder outputs the same number of time steps. The aligned tokens are then averaged over the sequence dimension and used as keys and values in a cross-attention layer with a single learnable query $Q\in\mathbb{R}^{1\times d}$, so the attention weights $O=\mathrm{Softmax}(QX^\top/\sqrt{d})X$ define a linear, input-dependent mixture of the four encoders' representations. This design keeps trainable overhead small and lets the LLM receive one compact token set per frame.

What would settle it

Re-run MERV and Video-LLaVA on Perception Test, TVQA, and the open-ended QA sets several times with the same GPT-3.5-turbo-0613 scorer and compute the variance across runs. If the 1.99-point Perception Test gap, and similarly sized gaps elsewhere, falls inside the run-to-run noise, the paper's advantage over its base model is not established.

Watch

Extended reading notes

Core claim

The central claim is that heterogeneous frozen visual encoders can be unified into a single additive video representation that inherits each encoder's specialty without sacrificing the others. MERV aligns the four encoders spatio-temporally: it picks input frame counts so temporal outputs match, adaptively average-pools the spatial grids to a common size, and linearly projects all channels into the LLM embedding dimension. A single learnable query then cross-attends over the averaged encoder tokens, so the attention weights form an input-dependent linear mixture of the four representations. The result, MERV, matches or beats every one of its single-encoder counterparts across open-ended and multiple-choice video QA benchmarks, including held-out sets, and on Something-Something v2 it captures both the temporal sensitivity of ViViT and the scene understanding of the contrastive encoders. Removing any of the four encoders degrades performance, which the paper takes as evidence that the fusion itself, not one dominant expert, carries the gains.

Load-bearing premise

The load-bearing premise is that the ChatGPT-based scoring protocol used for open-ended answers is stable enough that accuracy differences of roughly two to five points, reported from a single evaluation pass, reflect real model ability rather than evaluator noise.

Editorial extensions

If this is right

If the central claim holds, any LLaVA-style video model can be upgraded by adding frozen specialized encoders behind a spatio-temporally aligned projector, without redesigning the LLM or its training data.
Performance on benchmarks that emphasize temporal reasoning, such as Perception Test and TVQA, should keep improving as more complementary frozen encoders are added, provided the parallelization overhead stays small.
Because encoders remain frozen, new visual backbones can be swapped into MERV without retraining the language side from scratch, separating visual representation research from language-model training.
The Something-Something v2 analysis implies that video benchmarks should measure temporally reversed actions separately from general scene understanding, since single encoders differ sharply on those two axes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A consequence the authors leave implicit is that the cross-attention weights could serve as a cheap routing signal at inference time, letting a deployment skip encoders whose specialties are clearly irrelevant for a given video.
A testable extension is that the benefit of adding encoders should follow the diversity of their training objectives, not their number: adding a second video-language contrastive model added little, while the four-way mix of distinct paradigms gave the gains.
Another observable prediction is that models built this way should show their largest advantage on benchmarks that mix temporal and static reasoning in one dataset, because single encoders trade one off against the other.
Because the reported evaluation is a single GPT-scored pass, a natural check is whether these accuracy gaps persist under repeated scoring runs or under human agreement scoring.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Bottom line: this is a useful empirical paper that deserves a proper refereeing. MERV extends the multi-encoder fusion recipe from image LLMs to video, adding a temporal expert (ViViT) and spatio-temporal alignment, and shows consistent gains over single-encoder VideoLLMs on standard benchmarks. The ablations are thorough—projectors, token counts, fusion strategies, training recipes, encoder removal—and the held-out evaluation on NExT-QA, VLEP, TVQA is a real plus. The efficiency story (parallel encoders, similar step time) is credible and practically relevant.

The central claim that combining frozen specialized encoders helps holds up under the evidence. The same-recipe control in the appendix (Table 5) shows MERV with the Video-LLaVA recipe still beats Video-LLaVA, which addresses the main-table confound. And the encoder-removal ablation shows each of the four contributes something.

The soft spots are real but not fatal. The evaluation is dominated by single runs and ChatGPT-based scoring with no error bars, repeated evaluations, or seed sensitivity. On Perception Test, the headline gain over SeViLA relies on MERV (full); the frozen model is essentially tied with SeViLA. Within the ablations, channel-wise concatenation beats cross-attention on Perception Test (47.36 vs 46.21), and a 64 vs 100 token projection swings 0.78 points—both larger than some of the headline gains. So the specific fusion choice and the magnitude of advantages are not statistically secured. Also, the paper says in one place that code and weights are available, and in the appendix that they plan to release them; that needs fixing.

None of this breaks the paper. The recipe and the qualitative analysis are worth having. I'd send it to peer review and ask for variance estimates, repeated ChatGPT evaluations (or a human-agreement check), and a cleaner comparison that explicitly separates frozen/full and states which recipe each prior model used. The authors should also align the code-release statements.

Who is this for: anyone building VideoLLMs or thinking about multi-encoder fusion. It won't reorganize the field, but it's a practical step forward. A serious referee can help make the claims match the evidence.

Referee Report

4 major / 5 minor

Summary. The paper proposes MERV, a VideoLLM that fuses four frozen visual encoders (DINOv2, ViViT, SigLIP, LanguageBind) through spatio-temporal alignment, per-encoder linear projection, and a cross-attention mixer, before feeding the unified visual representation into LLaMA-2 7B. The authors fix the training data to match Video-LLaVA, introduce two training recipes (MERV frozen, a Stage-2-only recipe, and MERV full, which unfreezes the LLM during Stage 1), and evaluate on open-ended and multiple-choice video QA benchmarks, including three datasets held out during development. The main claims are that MERV outperforms Video-LLaVA by up to 4.62 points on TVQA, improves over SeViLA on zero-shot Perception Test by 2.2 points in the full recipe, and that the multi-encoder representation captures complementary specializations (spatial, temporal, and contrastive) with modest computational overhead. The appendix reports ablations of projectors, token counts, fusion strategies, training recipes, and per-encoder contributions, plus a Something-Something v2 analysis.

Significance. If the reported gains are statistically robust, MERV is a practically useful and conceptually clean contribution: it shows that combining existing frozen encoders can improve VideoLLMs without retraining the encoders, and it does this with a careful controlled comparison using Video-LLaVA's data mix, released code and weights, and a set of held-out benchmarks. The qualitative and quantitative analysis of encoder specializations (especially the temporal subset of Something-Something v2) is a strength, as is the systematic ablation of projectors, token budgets, and fusion strategies. The paper's main weakness is that the central numerical claims rest on single training runs and a single ChatGPT-based evaluation per answer, with no confidence intervals or repeated evaluations; several ablation differences are of the same magnitude as the headline gains. The significance of the method is therefore conditional on additional uncertainty quantification, but the core idea and the controlled experimental design are sound enough to warrant revision rather than rejection.

major comments (4)

[Section 4.1, Table 1; Appendix Tables 6b and 6c] The central comparison is reported without any measure of uncertainty. Each configuration appears to be a single training run, and open-ended answers are scored by a single ChatGPT call per answer (Section 4, 'Results using GPT-3.5-turbo for evaluation'), yet several load-bearing gaps are small: the Perception Test gain of MERV (frozen) over Video-LLaVA is 1.99 points, and the Perception Test advantage over SeViLA is 0.01 points for MERV (frozen) and 2.2 points only for MERV (full). The ablations show differences of comparable size, e.g. 0.78 points on Perception Test between 64 and 100 tokens (Appendix Table 6b) and channel concatenation beating the selected cross-attention fusion on Perception Test by 1.15 points (Appendix Table 6c). With single runs and no repeated ChatGPT scoring, the reader cannot determine whether the multi-encoder gains or the specific fusion choice are larger than run-to-run and evaluation noise. I ask for repeated evaluations (multiple seeds or at least repeated ChatGPT scoring with reported means and standard deviations, plus bootstrap confidence intervals on accuracy) for at least the main comparisons and the fusion ablation.
[Section 4.1, Table 1; Section 4.2.3; Section 4 (Dataset paragraph)] The claim of outperforming SeViLA by 2.2% on zero-shot Perception Test is not apples-to-apples. MERV (full) uses a different training recipe (Stage 1 with the LLM unfrozen) than SeViLA, and Perception Test is one of the five benchmarks explicitly described as used during development, not one of the three held-out benchmarks. With the default MERV (frozen) recipe, Perception Test accuracy is 46.21, essentially tied with SeViLA's 46.2, and on the other SeViLA-comparable benchmarks MERV (frozen) is below SeViLA on VLEP (58.66 vs 64.4) and NExT-QA (63.09 vs 63.6). The 'improve upon SeViLA by 2.2%' statement should be qualified to the full-recipe variant and to the development benchmark, and the generalization claim should rest primarily on the held-out results.
[Table 2c and Appendix Table 6c; Section 4.2.2] The selection of cross-attention as the final fusion module is not strongly supported by the reported numbers. Table 2c gives an average accuracy of 56.83 for cross-attention versus 56.64 for channel-wise concatenation, a 0.19-point difference over four development benchmarks, while Appendix Table 6c shows channel concatenation is better on Perception Test by 1.15 points (47.36 vs 46.21). The paper justifies the choice partly by the interpretability of attention weights, but the claimed performance advantage of the specific fusion mechanism over a simpler concatenation baseline appears to be within the unexplained single-run noise identified above. Please either provide uncertainty estimates that separate these configurations or soften the claim that cross-attention is the best-performing fusion strategy on accuracy.
[Section 4, Dataset paragraph; Table 1; Appendix Table 5] The headline held-out comparisons in Table 1 pair MERV (frozen), a Stage-2-only recipe, against Video-LLaVA, which uses the two-stage recipe. The same-recipe control in Appendix Table 5 partly addresses this concern, but only for the five development benchmarks; the three held-out benchmarks (NExT-QA, VLEP, TVQA) are not reported for the exact Video-LLaVA recipe, so the main held-out gains of +3.48 on NExT-QA and +4.62 on TVQA are not measured under a matched recipe. Reporting the held-out numbers for the same-recipe variant would strengthen the central claim that the multi-encoder representation, rather than the simplified training recipe, drives the improvement.

minor comments (5)

[Appendix A.2.2] The appendix says 'We plan to release our full code including training and evaluation as well as all model checkpoints for the camera-ready version,' while the footnote on page 2 states the code and pretrained weights are already available; please update the appendix to match the actual release status.
[Figures 10 and 13] The captions and sample text contain the typo 'iteracted' instead of 'interacted' in several places; please correct it.
[Table 1 and throughout] The inconsistent rendering of model names such as 'Video-LLaV A' and 'LLaMA-Vid' (with and without hyphens and spaces) should be unified to match the canonical names used by the cited papers.
[Section 4.2.3] The text says Stage 1 leaves 'approximately 1.3M vision-text pairs' unused; the cited Stage 1 data totals 558k image-text pairs plus 702k video-text pairs, which is roughly 1.26M, so the approximation should be stated consistently.
[Appendix A.3, Note on TGIF-QA] The note that Video-ChatGPT's and Video-LLaVA's author-reported TGIF numbers are on incomparable subsets is useful and should be moved or summarized in the main text near Table 1, since Table 1 reports TGIF numbers without this caveat.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MERV's gains are empirical benchmark comparisons, not reductions of outputs to inputs.

full rationale

MERV's central claim is an empirical architecture comparison: fusing four frozen encoders yields higher video QA accuracy than single-encoder baselines. The method is specified independently of the target benchmarks (encoder set, spatio-temporal alignment, pre-fusion projection, and cross-attention fusion are defined in Sections 3.1–3.2 and ablated in Section 4.2), and the reported held-out results on NExT-QA, VLEP, and TVQA are produced by a fixed trained pipeline evaluated once, not by fitting parameters to those datasets. No fitted quantity is renamed as a prediction, no self-citation carries a load-bearing argument, and no uniqueness theorem is imported from the authors' prior work. The development-set ablations are standard model selection; the lack of confidence intervals for ChatGPT-scored open-ended metrics is a statistical robustness concern, not a circularity concern. The derivation chain is therefore self-contained and empirically falsifiable.

Assumptions & free parameters 7 free parameters · 5 assumptions · 0 invented entities

MERV introduces no new physical entities; all added objects are ordinary learned modules (linear projectors, a cross-attention query) and pre-trained encoders drawn from prior work. The load-bearing degrees of freedom are the specific encoder set, the token compression, the fusion strategy, and the training recipe, all selected via development-set ablations. The main unstated assumptions concern the stability of the ChatGPT evaluator and the sufficiency of the fixed training data for aligning heterogeneous frozen features.

free parameters (7)

Number of encoders N=4 = 4
Chosen experimentally as the best set among tested options (Sec. 4.3); removing any encoder hurts average accuracy.
Pre-fusion output tokens per frame = 64
Ablation in Table 2b shows performance peaks at 64 tokens per frame on MSVD, MSRVTT, TGIF, and Perception Test.
Pre-fusion projector type = 2D adaptive average pooling
Table 2a shows 2D average pooling achieves the best average accuracy with zero trainable parameters; alternatives (attention, conv, 3D) are worse or much heavier.
Feature fusion strategy = Cross-attention with one learnable query
Table 2c shows cross-attention outperforms sequence/channel concatenation and fixed or learned scalar mixing in average accuracy.
Training recipe for MERV (frozen) = Stage 2 only, 1 epoch, lr 2e-5, batch 128, warmup 0.03, cosine schedule
Chosen as the default after ablations showed Stage-1 pretraining does not help much when only projector and fusion are trained (Appendix Table 5).
Frame sampling per encoder = 16 frames output t for all; ViViT takes 32 input frames
Default setting matching each encoder's preprocessing; an 8-to-16 frame increase improved average accuracy in Table 2a.
Specific encoder set (DINOv2, ViViT, SigLIP, LanguageBind) = DINOv2 ViT-L/14, ViViT-B/16x2, SigLIP ViT-B/16, LanguageBind ViT-L/14
Selected as the best among alternatives (CLIP, CLIP-ViP, V-JEPA, Hiera) based on overall performance; Hiera replacement and addition are ablated in Appendix A.5.1.

assumptions (5)

domain assumption Frozen pre-trained encoders provide complementary, linearly projectable representations that retain their specializations after average pooling.
The whole method assumes that simple spatial pooling and a linear layer do not destroy the specialized knowledge in each encoder. This is tested only indirectly through ablations and qualitative attention analysis (Sec. 4.2, 5.1).
domain assumption The LLaMA-2 7B backbone can reason effectively from the fused multi-encoder token sequence.
MERV feeds the LLM a single sequence of mixed visual tokens; the paper does not analyze whether the LLM fully exploits all encoder information, though cross-attention weights suggest it does (Sec. 5.1).
domain assumption The training data mix (LAION filtered captions, Valley video captions, LLaVA instructions, Video-ChatGPT instructions) is sufficient to align the multi-encoder representation with language.
The paper fixes the data mix to match Video-LLaVA and shows gains from that same data, but does not test whether more or different data would change the conclusions.
domain assumption The ChatGPT-based evaluator (gpt-3.5-turbo-0613) produces stable accuracy and score estimates.
Open-ended QA scores rely on a single evaluator call per answer, and no variance or repeat-run analysis is reported (Section 4, Evaluation).
domain assumption Standard benchmark datasets (MSVD, MSRVTT, TGIF, ActivityNet, NExT-QA, VLEP, TVQA, Perception Test) are reliable proxies for video understanding.
All conclusions about superiority rest on these benchmarks; the paper does not discuss benchmark noise or label quality.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Unifying Specialized Visual Encoders for Video Language Models." pith.science (2026). https://pith.science/paper/PZBKTLET

@misc{pith2026250101426,
  author       = {Pith},
  title        = {Pith review of: Unifying Specialized Visual Encoders for Video Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PZBKTLET}},
  note         = {Machine review of arXiv:2501.01426}
}

read the original abstract

The recent advent of Large Language Models (LLMs) has ushered sophisticated reasoning capabilities into the realm of video through Video Large Language Models (VideoLLMs). However, VideoLLMs currently rely on a single vision encoder for all of their visual processing, which limits the amount and type of visual information that can be conveyed to the LLM. Our method, MERV, Multi-Encoder Representation of Videos, instead leverages multiple frozen visual encoders to create a unified representation of a video, providing the VideoLLM with a comprehensive set of specialized visual knowledge. Spatio-temporally aligning the features from each encoder allows us to tackle a wider range of open-ended and multiple-choice video understanding questions and outperform prior state-of-the-art works. MERV is up to 3.7% better in accuracy than Video-LLaVA across the standard suite video understanding benchmarks, while also having a better Video-ChatGPT score. We also improve upon SeViLA, the previous best on zero-shot Perception Test accuracy, by 2.2%. MERV introduces minimal extra parameters and trains faster than equivalent single-encoder methods while parallelizing the visual processing. Finally, we provide qualitative evidence that MERV successfully captures domain knowledge from each of its encoders. Our results offer promising directions in utilizing multiple vision encoders for comprehensive video understanding.

Figures

Figures reproduced from arXiv: 2501.01426 by the authors.

**Figure 1.** Examples where a single encoder model is the only model to correctly answer Perception Test questions (Patr ˘ aucean ˘ et al., 2023), while MERV can correctly answer all types. language space, allowing for reasoning to happen in the highly expressive language domain. Most multimodal LLMs, such as LLaVA (Liu et al., 2023) for images and Video-LLaVA (Lin et al., 2024) for videos, opt for contrastively pre-trained enco… view at source ↗

**Figure 2.** MERV architecture and performance. (Left) MERV proceeds in three main stages. First, we feed in our input video into each of visual encoders to get different representations. They are then spatio-temporally aligned before being fused by a cross-attentive mixer. The output is a visual embedding with an additive mix of information from all the encoders, which is combined with the text query to generate the result. (Ri… view at source ↗

**Figure 3.** Extra encoders incur minimal step time overhead. We add encoders in the order of DINOv2, LanguageBind, SigLIP, ViViT, plotted alongside the slowest single encoder in each group. portant for optimal performance. Finally, we provide evidence for the efficiency of our method (ref [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (9 more)

**Figure 4.** Figure 4: Analysis plots supporting our design of multiple encoders, from their accuracy to their skill specializations. (a) Visual Encoder Subsets. MERV outperforms single-encoder VideoLLMs ( ), with our feature projectors unlocking more computational efficiency. Removing any …

**Figure 5.** Figure 5: Videos that give the highest attention weight for each of the encoders. The right-most column shows the average frame of the video. For more examples, see [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Single encoder vs. MERV on different types of video tasks. We plot the relative performance of VideoLLMs with different visual encoders. While each single encoder has its strength in different tasks, our method shows better performance than all the other single encode…

**Figure 8.** Figure 8: Samples of MERV in SSv2. Due to our design, our method shows better temporal action understanding than other VideoLLMs. (Top two rows) However, due to the difficulty of the task, we see failure cases for VideoLLMs. (Bottom two rows) 17 [PITH_FULL_IMAGE:figures/full_fi…

**Figure 9.** Figure 9: Example video of Something-Something V2. We see that ViViT show better performance in classes where temporal movement is critical for solving the task (Top row), while SigLIP performs better when the action can be inferred from the image without temporal information (B…

**Figure 10.** Figure 10: Example VideoLLM output on Something-Something v2. While SigLIP performs better on object and scene recognition, it fails to understand temporal actions. ViViT fails on the details of object recognition, but has better understanding in temporal movements. A.5. Additio…

**Figure 11.** Figure 11: Videos that give the highest attention weight for each of the encoders. The right-most column shows the average frame of the video. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_11.png]

**Figure 12.** Figure 12: Samples of MERV in video understanding. 21 [PITH_FULL_IMAGE:figures/full_fig_p021_12.png]

**Figure 13.** Figure 13: More qualitative results. MERV tends to show improved understanding in temporal-heavy videos as in Something-Something v2 dataset (Goyal et al., 2017) (Top Row), while retaining the performance on scenic understanding, seen from popular video benchmarks (Xu et al., 2…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Strefer: Empowering Video LLMs with Space-Time Referring and Reasoning via Synthetic Instruction Data
cs.CV 2025-09 conditional novelty 6.0 of 10

Adding Strefer's synthetic space-time reference questions to video instruction tuning improves mask-referred description/QA, timestamp QA, and temporal reasoning over a video-LLM baseline.
Exploring The Visual Feature Space for Multimodal Neural Decoding
cs.CV 2025-05 conditional novelty 5.0 of 10

VINDEX decodes fMRI into nested 9-token CLIP features that feed a frozen MLLM, improving detailed brain captioning and QA over a regression baseline, with a new benchmark.

Reference graph

Works this paper leans on

14 extracted references · 12 canonical work pages · cited by 2 Pith papers

[2]

Liu, H., Li, C., Li, Y ., Li, B., Zhang, Y ., Shen, S., and Lee, Y . J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/ . Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating wikipedia by summarizing long sequences. In ICLR,

work page 2024
[3]

Visual Encoder and LLM Information. Model Visual Encoder LLM Video-Chat (Li et al., 2023c) ViT-G (EV A-CLIP) (Sun et al., 2023b)StableVicuna (contributors, 2023)LLaMA-Adapter (Zhang et al., 2024b) CLIP (Radford et al.,

work page 2023
[4]

Fine-grained audio-visual joint representations for multimodal large language models

Sun, G., Yu, W., Tang, C., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. Fine-grained audio-visual joint representations for multimodal large language models. arXiv, 2023a. Sun, Q., Fang, Y ., Wu, L., Wang, X., and Cao, Y . Eva- clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023b. Tong, S., Brown, E., Wu...

arXiv
[5]

We explore different training recipe strategies, starting with the standard LLaV A recipe which Video-LLaV A adopted, along with some other variations

Ablation of training stage recipes. We explore different training recipe strategies, starting with the standard LLaV A recipe which Video-LLaV A adopted, along with some other variations. Methods MSVD-QAMSRVTT-QATGIF-QAPerception TestActivityNet-QA Acc Score Acc Score Acc Score Acc Acc Score MERV (frozen) 70.97 3.76 59.03 3.25 51.1 3.26 46.21 50.87 3.34 M...

work page 2024
[7]

Vicuna-7B v1.1 (Chiang et al., 2023)SeViLA (Yu et al., 2024b)ViT-G (EV A-CLIP) (Sun et al., 2023b) + BLIP-2 Q-Former (Li et al., 2023b)FlanT5-XL (3B) (Chung et al., 2024)LLaMA-VID-7B (Li et al., 2024b) EV A-G (Fang et al.,

work page 2023
[8]

Vicuna-7B v1.5 (Chiang et al., 2023)LLaMA-VID-13B (Li et al., 2024b) EV A-G (Fang et al.,

work page 2023
[9]

Vicuna-13B v1.5 (Chiang et al., 2023)Video-LLaV A* (Lin et al.,

work page 2023
[10]

Performance on Something-Something V2 - OpenEnded. These are the performance in shown Figure 4b MERV MERV-FullLanguageBind DinoV2 ViViT SigLIPLLaMA-Vid-7B LLaMA-Vid-13B VideoLLaV A Smth-Smth V2-OE-Temporal6.82 9.13 3.63 3.88 5.50 4.25 6.07 3.94 5.57 Smth-Smth V2-OE 17.70 20.65 13.83 11.03 10.53 13.84 16.47 15.62 19.18 Smth-Smth V2-MCQ-Temporal36.84 40.65 ...

work page

Show all 14 references

[11]

Performance with Hiera (Ryali et al., 2023). Methods MSVD-QA MSRVTT-QA TGIF-QA Perception ActivityNet-QA Acc Score Acc Score Acc Score Acc Acc Score VideoLLaV A 67.74 3.69 56.90 3.18 47.99 3.17 47.08 3.27 44.22 MERV 70.97 3.76 59.03 3.25 51.1 3.26 50.87 3.34 46.21 MERV , ViViT...

2023
[12]

While SigLIP performs better on object and scene recognition, it fails to understand temporal actions

Example VideoLLM output on Something-Something v2. While SigLIP performs better on object and scene recognition, it fails to understand temporal actions. ViViT fails on the details of object recognition, but has better understanding in temporal movements. A.5. Additional Exper...

2023
[2017]

(Top Row), while retaining the performance on scenic understanding, seen from popular video bench- marks (Xu et al., 2017; Yu et al.,

2017
[2021]

LLaMA-1 7B (Touvron et al., 2023a)Video-LLaMA (Zhang et al., 2023)ViT-G (EV A-CLIP) (Sun et al., 2023b) + BLIP-2 Q-Former (Li et al., 2023b)Vicuna-7B v0 (Chiang et al., 2023)Video-ChatGPT (Maaz et al.,

2023
[2023]

Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H

URL https://lmsys.org/blog/ 2023-03-30-vicuna/ . Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research,

2023
[2024]

Gpt-4 technical report

OpenAI, R. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5),

arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [2]

Liu, H., Li, C., Li, Y ., Li, B., Zhang, Y ., Shen, S., and Lee, Y . J. Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https://llava-vl.github.io/blog/ 2024-01-30-llava-next/ . Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating wikipedia by summarizing long sequences. In ICLR,

work page 2024

[2] [3]

Visual Encoder and LLM Information. Model Visual Encoder LLM Video-Chat (Li et al., 2023c) ViT-G (EV A-CLIP) (Sun et al., 2023b)StableVicuna (contributors, 2023)LLaMA-Adapter (Zhang et al., 2024b) CLIP (Radford et al.,

work page 2023

[3] [4]

Fine-grained audio-visual joint representations for multimodal large language models

Sun, G., Yu, W., Tang, C., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., and Zhang, C. Fine-grained audio-visual joint representations for multimodal large language models. arXiv, 2023a. Sun, Q., Fang, Y ., Wu, L., Wang, X., and Cao, Y . Eva- clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389, 2023b. Tong, S., Brown, E., Wu...

arXiv

[4] [5]

We explore different training recipe strategies, starting with the standard LLaV A recipe which Video-LLaV A adopted, along with some other variations

Ablation of training stage recipes. We explore different training recipe strategies, starting with the standard LLaV A recipe which Video-LLaV A adopted, along with some other variations. Methods MSVD-QAMSRVTT-QATGIF-QAPerception TestActivityNet-QA Acc Score Acc Score Acc Score Acc Acc Score MERV (frozen) 70.97 3.76 59.03 3.25 51.1 3.26 46.21 50.87 3.34 M...

work page 2024

[5] [7]

Vicuna-7B v1.1 (Chiang et al., 2023)SeViLA (Yu et al., 2024b)ViT-G (EV A-CLIP) (Sun et al., 2023b) + BLIP-2 Q-Former (Li et al., 2023b)FlanT5-XL (3B) (Chung et al., 2024)LLaMA-VID-7B (Li et al., 2024b) EV A-G (Fang et al.,

work page 2023

[6] [8]

Vicuna-7B v1.5 (Chiang et al., 2023)LLaMA-VID-13B (Li et al., 2024b) EV A-G (Fang et al.,

work page 2023

[7] [9]

Vicuna-13B v1.5 (Chiang et al., 2023)Video-LLaV A* (Lin et al.,

work page 2023

[8] [10]

Performance on Something-Something V2 - OpenEnded. These are the performance in shown Figure 4b MERV MERV-FullLanguageBind DinoV2 ViViT SigLIPLLaMA-Vid-7B LLaMA-Vid-13B VideoLLaV A Smth-Smth V2-OE-Temporal6.82 9.13 3.63 3.88 5.50 4.25 6.07 3.94 5.57 Smth-Smth V2-OE 17.70 20.65 13.83 11.03 10.53 13.84 16.47 15.62 19.18 Smth-Smth V2-MCQ-Temporal36.84 40.65 ...

work page