Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Chenzhuang Du; Chonghua Liao; Hang Zhao; Jiacheng You; Jie Fu; Yue Zhao

RECORD

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Chenzhuang Du, Yue Zhao, Chonghua Liao, Jiacheng You, Jie Fu, Hang Zhao

Computer Vision and Pattern Recognition · Submitted 8 Oct 2023

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim By freezing the weights of uni-modal fine-tuned models, adding extra trainable rank decomposition matrices to them, and subsequently performing multi-modal joint training, our method enhances adaptation between modalities and boosts overall performance.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2310.05193 (version 1)

Pith IDLYK6R66Y, Pith's permanent record of this paper and its review

CitationChenzhuang Du, Chonghua Liao, Hang Zhao, Jiacheng You, Jie Fu, Yue Zhao. "Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models." arXiv:2310.05193 (2023). https://pith.science/paper/LYK6R66Y

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,339 input tokens.

What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.16 · human participants 0.09 · controlled comparison 0.44 · randomized assignment 0.01 · states quantitative result 0.04 · compares with prior results 0.79 · mentions code or data availability 0.02.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; show every role.

This paper investigates how to better leverage large-scale pre-trained uni-modal models to further enhance discriminative multi-modal learning. Even when fine-tuned with only uni-modal data, these models can outperform previous multi-modal models in certain tasks. It's clear that their incorporation into multi-modal learning would significantly improve performance. However, multi-modal learning with these models still suffers from insufficient learning of uni-modal features, which weakens the resulting multi-modal model's generalization ability. While fine-tuning uni-modal models separately and then aggregating their predictions is straightforward, it doesn't allow for adequate adaptation between modalities, also leading to sub-optimal results. To this end, we introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA). By freezing the weights of uni-modal fine-tuned models, adding extra trainable rank decomposition matrices to them, and subsequently performing multi-modal joint training, our method enhances adaptation between modalities and boosts overall performance. We demonstrate the effectiveness of MMLoRA on three dataset categories: audio-visual (e.g., AVE, Kinetics-Sound, CREMA-D), vision-language (e.g., MM-IMDB, UPMC Food101), and RGB-Optical Flow (UCF101).

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	This paper investigates how to better leverage large-scale pre-trained uni-modal models to further enhance discriminative multi-modal learning.	contribution 0.57	0.43	0.06	asserted 0.61
s1	Even when fine-tuned with only uni-modal data, these models can outperform previous multi-modal models in certain tasks.	background 0.91	0.23	0.06	asserted 0.76
s2	It's clear that their incorporation into multi-modal learning would significantly improve performance.	background 0.65	0.11	0.15	asserted 0.99
s3	However, multi-modal learning with these models still suffers from insufficient learning of uni-modal features, which weakens the resulting multi-modal model's generalization ability.	gap 0.83	0.21	0.06	asserted 1.00
s4	While fine-tuning uni-modal models separately and then aggregating their predictions is straightforward, it doesn't allow for adequate adaptation between modalities, also leading to sub-optimal results.	gap 0.56	0.16	0.03	asserted 1.00
s5	To this end, we introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA).	contribution 0.95	0.84	0.06	asserted 0.97
s6	By freezing the weights of uni-modal fine-tuned models, adding extra trainable rank decomposition matrices to them, and subsequently performing multi-modal joint training, our method enhances adaptation between modalities and boosts overall performance.	method 0.97	0.83	0.26	asserted 1.00
s7	We demonstrate the effectiveness of MMLoRA on three dataset categories: audio-visual (e.g., AVE, Kinetics-Sound, CREMA-D), vision-language (e.g., MM-IMDB, UPMC Food101), and RGB-Optical Flow (UCF101).	result 0.42	0.95	0.17	asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97.

What was examined, and what was not

sources read

arXiv listing abstract v1

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

This paper investigates how to better leverage large-scale pre-trained uni-modal models to further enhance discriminative multi-modal learning. Even when fine-tuned with only uni-modal data, these models can outperform previous multi-modal models in certain tasks. It's clear that their incorporation into multi-modal learning would significantly improve performance. However, multi-modal learning with these models still suffers from insufficient learning of uni-modal features, which weakens the resulting multi-modal model's generalization ability. While fine-tuning uni-modal models separately and then aggregating their predictions is straightforward, it doesn't allow for adequate adaptation between modalities, also leading to sub-optimal results. To this end, we introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA). By freezing the weights of uni-modal fine-tuned models, adding extra trainable rank decomposition matrices to them, and subsequently performing multi-modal joint training, our method enhances adaptation between modalities and boosts overall performance. We demonstrate the effectiveness of MMLoRA on three dataset categories: audio-visual (e.g., AVE, Kinetics-Sound, CREMA-D), vision-language (e.g., MM-IMDB, UPMC Food101), and RGB-Optical Flow (UCF101).

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 25 sentences without a confident label

Uses the method or tool of 3 papers

When fine-tuning the language model in a uni-modal setting, we use BertAdam [ref] as the optimizer; We use SGD when training the optical flow in UCF101; For all other experiments, we employ AdamW [ref] .

Decoupled Weight Decay Regularization · Experimental Settings · 1.00
When fine-tuning the language model in a uni-modal setting, we use BertAdam [ref] as the optimizer; We use SGD when training the optical flow in UCF101; For all other experiments, we employ AdamW [ref] .

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding · Experimental Settings · 1.00
Our audio preprocessing is consistent with that of [ref] or [ref] ; (3) For text, we use the corresponding tokenizer for different language models.

AST: Audio Spectrogram Transformer · Experimental Settings · 0.97

Uses data or values from 1 paper

The UCF101 dataset, presented by [ref] , is designed for action recognition, featuring 101 distinct action categories.

UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild · Datasets · 0.84

Compares with 2 papers

Here, we compare MMLoRA with MBT [ref] , MMBT [ref] , and PMF [ref] .

Supervised Multimodal Bitransformers for Classifying Images and Text · Main Results of MMLoRA · 1.00
The methods compared include balancing training by adding loss (G-Blending [ref] , PMR [ref] , UMT [ref] ), adjusting the training progress of different modalities by modifying gradients (OGM-GE [ref] ), novel fusion framework (MBT [ref] ),or directly averaging uni-modal predictions (UME [ref] ).

On Uni-Modal Feature Learning in Supervised Multi-Modal Learning · Main Results of MMLoRA · 0.98

Cites as support 1 paper

As can be seen from Section 3.2 of the paper [ref] , in multi-modal datasets, different modalities need to adapt to each other to produce better results.

On Uni-Modal Feature Learning in Supervised Multi-Modal Learning · Multi-Modal Low-Rank Adaptation learning (MMLoRA) · 0.97

Cites as background 9 papers

Parameter-Efficient Fine-Tuning is a methodology designed to fine-tune only a subset of parameters in large language models, enhancing their adaptability to downstream tasks [ref] .

QLoRA: Efficient Finetuning of Quantized LLMs · Related Work · 0.99
Parameter-Efficient Fine-Tuning is a methodology designed to fine-tune only a subset of parameters in large language models, enhancing their adaptability to downstream tasks [ref] .

LoRA: Low-Rank Adaptation of Large Language Models · Related Work · 0.99
Parameter-Efficient Fine-Tuning is a methodology designed to fine-tune only a subset of parameters in large language models, enhancing their adaptability to downstream tasks [ref] .

P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks · Related Work · 0.99

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.