UnIVAL: Unified Model for Image, Video, Audio and Language Tasks

Alexandre Rame; Corentin Dancette; Matthieu Cord; Mustafa Shukor

RECORD

UnIVAL: Unified Model for Image, Video, Audio and Language Tasks

Mustafa Shukor, Corentin Dancette, Alexandre Rame, Matthieu Cord

Computer Vision and Pattern Recognition · Submitted 30 Jul 2023

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2307.16184 (version 2)

Pith ID72MGGVU7, Pith's permanent record of this paper and its review

CitationAlexandre Rame, Corentin Dancette, Matthieu Cord, Mustafa Shukor. "UnIVAL: Unified Model for Image, Video, Audio and Language Tasks." arXiv:2307.16184 (2023). https://pith.science/paper/72MGGVU7

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 9,545 input tokens.

What the abstract itself states, as probabilities: claims first 0.11 · states limitation 0.62 · human participants 0.03 · controlled comparison 0.32 · randomized assignment 0.01 · states quantitative result 0.88 · compares with prior results 0.94 · mentions code or data availability 0.99.

The abstract, sentence by sentence

arXiv listing abstract v2. Results and the main claim are underlined; show every role.

Large Language Models (LLMs) have made the ambitious quest for generalist agents significantly far from being a fantasy. A key hurdle for building such general models is the diversity and heterogeneity of tasks and modalities. A promising solution is unification, allowing the support of a myriad of tasks and modalities within one unified framework. While few large models (e.g., Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more than two modalities, current small to mid-scale unified models are still limited to 2 modalities, usually image-text or video-text. The question that we ask is: is it possible to build efficiently a unified model that can support all modalities? To answer this, we propose UnIVAL, a step further towards this ambitious goal. Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model. Our model is efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning. UnIVAL shows competitive performance to existing state-of-the-art approaches, across image and video-text tasks. The feature representations learned from image and video-text modalities, allows the model to achieve competitive performance when finetuned on audio-text tasks, despite not being pretrained on audio. Thanks to the unified model, we propose a novel study on multimodal model merging via weight interpolation of models trained on different multimodal tasks, showing their benefits in particular for out-of-distribution generalization. Finally, we motivate unification by showing the synergy between tasks. The model weights and code are released here: https://github.com/mshukor/UnIVAL.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Large Language Models (LLMs) have made the ambitious quest for generalist agents significantly far from being a fantasy.	background 1.00	0.04	0.03	asserted 0.99
s1	A key hurdle for building such general models is the diversity and heterogeneity of tasks and modalities.	background 0.89	0.07	0.02	asserted 1.00
s2	A promising solution is unification, allowing the support of a myriad of tasks and modalities within one unified framework.	background 0.99	0.04	0.07	hedged possibility 0.55
s3	While few large models (e.g., Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more than two modalities, current small to mid-scale unified models are still limited to 2 modalities, usually image-text or video-text.	gap 0.57	0.07	0.04	asserted 0.99
s4	The question that we ask is: is it possible to build efficiently a unified model that can support all modalities?	gap 0.92	0.06	0.04	hedged possibility 0.62
s5	To answer this, we propose UnIVAL, a step further towards this ambitious goal.	contribution 0.99	0.69	0.08	asserted 0.74
s6	Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model.	contribution 0.88	0.95	0.46	asserted 1.00
s7	Our model is efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning.	method 0.99	0.93	0.06	asserted 1.00
s8	UnIVAL shows competitive performance to existing state-of-the-art approaches, across image and video-text tasks.	result 1.00	0.93	0.17	asserted 1.00
s9	The feature representations learned from image and video-text modalities, allows the model to achieve competitive performance when finetuned on audio-text tasks, despite not being pretrained on audio.	result 0.97	0.93	0.13	asserted 0.99
s10	Thanks to the unified model, we propose a novel study on multimodal model merging via weight interpolation of models trained on different multimodal tasks, showing their benefits in particular for out-of-distribution generalization.	contribution 0.99	0.94	0.10	asserted 0.99
s11	Finally, we motivate unification by showing the synergy between tasks.	interpretation 0.67	0.78	0.22	asserted 0.97
s12	The model weights and code are released here: https://github.com/mshukor/UnIVAL.	contribution 0.86	0.52	0.02	asserted 0.84

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.95.

What was examined, and what was not

sources read

arXiv listing abstract v2

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Large Language Models (LLMs) have made the ambitious quest for generalist agents significantly far from being a fantasy. A key hurdle for building such general models is the diversity and heterogeneity of tasks and modalities. A promising solution is unification, allowing the support of a myriad of tasks and modalities within one unified framework. While few large models (e.g., Flamingo (Alayrac et al., 2022), trained on massive datasets, can support more than two modalities, current small to mid-scale unified models are still limited to 2 modalities, usually image-text or video-text. The question that we ask is: is it possible to build efficiently a unified model that can support all modalities? To answer this, we propose UnIVAL, a step further towards this ambitious goal. Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model. Our model is efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning. UnIVAL shows competitive performance to existing state-of-the-art approaches, across image and video-text tasks. The feature representations learned from image and video-text modalities, allows the model to achieve competitive performance when finetuned on audio-text tasks, despite not being pretrained on audio. Thanks to the unified model, we propose a novel study on multimodal model merging via weight interpolation of models trained on different multimodal tasks, showing their benefits in particular for out-of-distribution generalization. Finally, we motivate unification by showing the synergy between tasks. The model weights and code are released here: https://github.com/mshukor/UnIVAL.

Human review

How Pith checks papers → · Full audit of this paper →

References

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Harmony: A Unified Framework for Modality Incremental Learning
cs.LG 2025-04 conditional novelty 6.0 of 10

Harmony is a framework that lets a single model learn new data modalities sequentially, using only unimodal data per phase, and outperforms continual learning baselines on two action recognition benchmarks.
MM-VID: Advancing Video Understanding with GPT-4V(ision)
cs.CV 2023-10 conditional novelty 4.0 of 10

A GPT-4V plus GPT-4 pipeline converts hour-long videos into timestamped text scripts, enabling grounded question answering and auto-generated audio description that a nine-person user study rates close to human-writte...
Vision Generalist Model: A Survey
cs.CV 2025-06 conditional novelty 3.0 of 10

A structured review of vision generalist models, classifying them into encoding-based and sequence-to-sequence frameworks and summarizing datasets, benchmarks, techniques, and open problems.

How the literature uses it

14 arXiv papers cite this in 21 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 9 sentences without a confident label

Cites it as support 1 paper

This is becoming increasingly important as recent works [ref] , in preliminary investigations, have highlighted qualitatively several major flaws (e.g., hallucinations), showing that LMMs are still not aligned with the needs for deployment in real-world applications.

Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning · Introduction · 0.99

Cites it as background 9 papers

Prior studies have explored this direction from several perspectives, including merging models trained on different modalities or multimodal tasks, composing existing MLLMs for capability enhancement or modality expansion [ref] , and developing more adaptive or robust post-finetuning merging strategies [ref] .

PivotMerge: Bridging Heterogeneous Multimodal Pre-training via Post-Alignment Model Merging · Introduction · 0.88
Recently, several multi-model and LLM based models capable of audio understanding and question answering have emerged such as LLaMA Adapter [ref] , UniVAL [ref] and LTU [ref] .

Music Understanding LLaMA: Advancing Text-to-Music Generation with Question Answering and Captioning · Introduction · 0.88
These methods first transform various modalities and tasks into sequences or sets of discrete tokens [ref] , and then train a single Transformer on these tokens using either a sequence modeling [ref] or masked modeling objective [ref] .

4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities · Related Work · 0.91

What it cites · 26 sentences without a confident label

Contests 1 paper

Contrary to previous work [ref] , we find it more beneficial to balance the tasks in the batch, especially when using highly unbalanced datasets.

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework · Efficient pretraining · contest 0.74

Uses the method or tool of 4 papers

We follow [ref] suggesting that averaging networks in weights can combine their abilities without any computational overhead.

Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Models · Related Work · 0.87
We follow [ref] suggesting that averaging networks in weights can combine their abilities without any computational overhead.

Elastic Weight Removal for Faithful and Abstractive Dialogue Generation · Related Work · 0.87
The modality-specific encoders are ResNet-101 pretrained on ImageNet as image encoder, 3D ResNext-101 [ref] pretrained on kinetics 400 as video encoder and PANN encoder pretrained for audio classification as audio encoder, we do not skip the last block as done by previous approaches [ref] .

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework · Efficient pretraining · 0.84

Compares with 4 papers

Despite pretraining on less data for less number of steps, our approach is on par with the previous unified model OFA [ref] finetuned from the author's released checkpoint ($OFA_Base^†$).

OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework · Image-text tasks · 0.99
The closest works to us are indeed OFA [ref] and Unified-IO [ref] , however, we propose to unify tasks across more modalities, with significantly smaller model and dataset sizes.

Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks · Related Work · 0.99
Compared to the previous OmniVL model [ref] that pretrain on both image and video text datasets, we largely outperform it by more than 3 points CIDEr.

OmniVL:One Foundation Model for Image-Language and Video-Language Tasks · Image-text tasks · 1.00

Cites as support 2 papers

To tackle multimodal tasks at small to mid-scale, we employ an encoder-decoder LM, as its effectiveness for multimodal tasks has been demonstrated compared to decoder-only LMs [ref] , in addition to its superiority in zero-shot generalization after multitask training [ref] .

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision · Unified model · 0.98
Previous studies [ref] have demonstrated notable performance enhancements when employing this paradigm for shared visual encoders (applied to both images and videos).

OmniVL:One Foundation Model for Image-Language and Video-Language Tasks · Efficient pretraining · 0.98

Cites as background 8 papers

More data-efficient approaches [ref] relied on additional multimodal interaction modules and variety of training objectives such as image-text matching, masked language modeling and image-text contrastive [ref] .

An Empirical Study of Training End-to-End Vision-and-Language Transformers · Related Work · 0.92
More data-efficient approaches [ref] relied on additional multimodal interaction modules and variety of training objectives such as image-text matching, masked language modeling and image-text contrastive [ref] .

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation · Related Work · 0.92
In the video-language community, similar approaches have tried to model the interaction between language and frames sequences [ref] .

VindLU: A Recipe for Effective Video-and-Language Pretraining · Related Work · 0.96

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.