Pith. sign in

REVIEW 6 major objections 6 minor 141 references

MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping

T0 review · 6 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Grouping multimodal instruction-tuning tasks by the type of text–image interaction — redundant, unique, or synergistic — improves accuracy by up to 26.7% over unselective mixing.

desk verdict Promising grouping idea undercut by missing core measurement and evaluation leakage; worth a referee but needs revision. read the letter →

arxiv 2506.02308 v3 pith:XWO324FU submitted 2025-06-02 cs.LG cs.AI

Xiaojun Shan , Qi Cao , Xing Han , Haofei Yu , Paul Pu Liang This is my paper
classification cs.LGcs.AI
keywords multimodalinstructiontuningtaskgroupinginteractionredundancyuniquenesssynergyLoRAfine-tuningQwen2-VL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the key to effective multimodal instruction tuning is not adding more tasks but grouping tasks by how their two modalities interact: whether the text and image mostly repeat each other (redundancy), each contribute unique pieces of information (uniqueness), or only make sense when fused (synergy). It introduces a dataset-level Multimodal Interaction score that measures how closely a unimodal model's predictions agree with a full multimodal model's predictions on a dataset, then groups tasks whose scores are close. Fine-tuning Qwen2-VL (7B) separately on each interaction group, via LoRA, beats single-task tuning, unselective mixing of all tasks, and two prior task-grouping baselines, with reported gains as large as 26.7% over unselective mixing on HEMM-derived benchmarks. The paper takes this as evidence that interaction-aware grouping lets a model practice a shared cross-modal skill within a group while avoiding negative transfer between mismatched tasks.

What carries the argument

The central object is the Multimodal Interaction score, defined in Eq. (1): it averages, over draws of $C$ instances, the semantic agreement $\delta$ between each unimodal model's prediction and the multimodal model's prediction on the same instance. The Multimodal Dataset Distance between two datasets is the absolute difference of their interaction scores, $d(D_A, D_B) = |\Delta_{1,2}(D_A) - \Delta_{1,2}(D_B)|$. The grouping step places tasks with similar scores into one of three interaction groups (redundancy, uniqueness, synergy), and a separate instruction-tuned model is trained per group. The mechanism doing the work is that tasks within a group share the same cross-modal computation, so jointly fine-tuning them rehearses the same skill — cross-modal fusion for synergy, selective attention for uniqueness, redundant cross-checking for redundancy — while batches from different interaction types no longer interfere.

What would settle it

Recompute $\Delta_{1,2}(D)$ for the 18 training datasets using the paper's unimodal and multimodal predictions and a fixed semantic similarity $\delta$; if the resulting redundancy/uniqueness/synergy partition does not reproduce the paper's Table 1 labels, then the reported gains are attributable to the labels rather than to the interaction score.

Watch

Extended reading notes

Core claim

The central discovery is that a scalar dataset-level interaction score, $\Delta_{1,2}(D) = \frac{1}{S} \sum_s \frac{1}{C} \sum_j [\delta(y_{1,j}, y_{m,j}) + \delta(y_{2,j}, y_{m,j})]$, computed as the average semantic agreement between each unimodal model and the multimodal model, is a useful organizing principle for instruction-tuning data. Datasets with scores near 2 are redundancy-dominant, near 1 uniqueness-dominant, and near 0 synergy-dominant. Grouping tasks by this score — equivalently by small pairwise dataset distance $d(D_A, D_B) = |\Delta_{1,2}(D_A) - \Delta_{1,2}(D_B)|$ — and fine-tuning a separate model per group yields consistent accuracy gains on the group's tasks, outperforming single-task fine-tuning, unselective multi-task fine-tuning, INSTA's instruction-similarity clustering, and MixLoRA's conditional mixture-of-LoRA routing.

Load-bearing premise

The grouping labels were actually computed from Eq. (1) with the stated unimodal and multimodal models, not assigned by hand, and every grouped task has a meaningful text modality for the unimodal text model.

Editorial extensions

If this is right

  • Adding more instruction-tuning tasks without grouping can hurt rather than help; the paper shows unselective multi-task fine-tuning underperforms single-task fine-tuning on several HEMM benchmarks.
  • Grouping by interaction type beats surface-level similarity, as MINT outperforms INSTA's instruction-text similarity clustering on nearly every test dataset reported in Table 2.
  • Interaction-aware grouping also beats architectural solutions, outperforming MixLoRA's conditional mixture-of-LoRA routing while keeping the fine-tuning pipeline simple and adding no gating parameters.
  • The grouped model generalizes to unseen tasks within the same interaction type, as seen on MagicBrush, which appears in none of the training groups yet is handled best by the synergy-tuned model, suggesting the grouping learns interaction-specific skills that transfer.
  • The method yields per-group specialized models $\Theta_R$, $\Theta_U$, $\Theta_S$, showing that specialization within a group and generalization across domains are compatible when tasks share an interaction type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the interaction score is robust to the choice of unimodal and multimodal models, the same grouping could apply to other modality pairs (audio-text, video-text) and other base models, since the score is model-family agnostic; the paper demonstrates it only with Qwen2-VL 7B.
  • The reported gains are measured on in-group test sets; a stress test would be whether interaction-grouped tuning generalizes to an unseen task of a different interaction type than any training group, since the paper trains no cross-group model.
  • The MI score compares whole predicted outputs, so a finer-grained token- or span-level semantic similarity might separate tasks that currently have similar scores but differ in which parts of the input carry the signal, potentially refining the grouping.
  • Because the paper excludes most VQA-only benchmarks and some sources likely seen during pre-training, the measured advantage may partly reflect curation choices; whether interaction grouping adds value on top of aggressive dataset filtering is untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes MINT, a task-grouping strategy for multimodal instruction tuning. The method defines a dataset-level 'multimodal interaction' score (Eq. 1) as the average semantic agreement between two unimodal predictors and a multimodal predictor, categorizes datasets into redundancy, uniqueness, or synergy groups, and fine-tunes Qwen2-VL separately on each group with LoRA. The authors report that this grouping outperforms single-task tuning, unselective multi-task tuning, INSTA, and MixLoRA on HEMM-derived benchmarks, with gains up to 26.7% over the base model.

Significance. The core idea—grouping instruction-tuning tasks by the type of cross-modal interaction rather than surface similarity—is appealing and, if substantiated, would be a practical and interpretable design principle. The paper also makes a strong falsifiable prediction: interaction-based grouping beats similarity-based grouping and unselective mixing. However, the current manuscript does not demonstrate that its groupings actually follow from the proposed computation, and the experimental protocol has leaks; consequently the central claim is not yet established.

major comments (6)
  1. [Sec. 3.1, Eq. (1); Sec. 4.1.1] The paper never reports the computed MI scores that are supposed to determine the RUS groups. None of the ingredients in Eq. (1) are specified: the unimodal models f1 and f2, the multimodal model fm, the similarity function δ, the number of draws S, the sample size C, or the resulting ∆1,2(D) values for the 18 datasets. Section 4.1.1 states that the groups were obtained 'after computing the Multimodal Dataset Distance,' but no numerical output is shown. Since the central empirical claim is that grouping by the computed MI score yields the reported gains, this omission is load-bearing: if the labels were assigned by hand or with different model choices, Table 2 evaluates those labels rather than the proposed method. The authors must report the exact computation and the resulting scores for every dataset.
  2. [Sec. 3.1, Eq. (1); Table 1] For at least five selected tasks (FER-2013, RESISC45, UCMERCED, INATURALIST, ENRICO), the 'text' modality is a fixed prompt with no task-dependent content. A text-only model f2 receiving only that prompt has no information about the image class, so δ(y2, ym) in Eq. (1) reflects model priors and answer-space statistics, not a property of the multimodal interaction. These five tasks are all assigned to the Uniqueness group, which suggests that the uniqueness labels may be an artifact of a degenerate text modality. The authors need to either define a meaningful task-conditioned text input for these datasets or provide evidence that the MI computation is not degenerate; without this, the claimed interaction-based grouping is not supported for the Uniqueness group.
  3. [Sec. 4.1.1; Sec. 4.2, Table 2] The experimental protocol does not support the claimed generalization. The training data are filtered by post hoc criteria—'Most VQA–only benchmarks' and sources 'likely seen during Qwen-VL2's pretraining'—but no list of excluded datasets is given. More seriously, at least 8 of the 12 test tasks (SLAKE, PATHVQA, NLVR, HATEFULMEMES, NYCARTOON, SCIENCEQA, INATURALIST, UCMERCED) also appear in the training groups, so for these tasks the comparison measures in-distribution fine-tuning, not generalization. The statement that test tasks are 'seen in the training set, like SLAKE, or unseen ... like MagicBrush' is accurate for a few tasks, but the reported averages in Figure 4 and Table 2 mix seen and unseen tasks. Please report seen and unseen results separately and specify exactly which HEMM tasks were excluded and why.
  4. [Sec. 4.2, Table 2] No variance or seed information is reported for any of the fine-tuning runs. Every entry in Table 2 and every point in Figures 3–5 appears to be a single run, so the reader cannot assess whether the differences between MINT and the baselines (which in some rows are only a few points) are statistically meaningful. At a minimum, the paper should report the mean and standard deviation over at least three seeds, and ideally a paired test across the test tasks.
  5. [Sec. 4.1.1 vs Table 1] The training groups in Section 4.1.1 are inconsistent with Table 1: the Redundancy group lists OK-VQA, which is not in Table 1, while Table 1 lists IRFL as Synergy but IRFL does not appear in any training group. This prevents the reader from reconstructing the exact training mixture. Please align the dataset table with the actual splits used in the experiments.
  6. [Appendix D] The model details in Appendix D describe a different evaluation setup than the main text: it states that Qwen2-VL is evaluated at 1.8B parameters with a link to Qwen-VL, whereas the main experiments use Qwen2-VL 7B (Section 4.1.2). The sentence 'All the models except for Gemini and GPT-4V are open source' is also inconsistent with the list that follows, which contains no Gemini or GPT-4V entries. Please correct the appendix to match the models actually evaluated.
minor comments (6)
  1. [Sec. 3.3, Step 2] The definition of Multimodal Dataset Distance in Eq. (2) is the absolute difference of MI scores, and the partition P_RUS is formed by the MI categories; the claim that tasks are grouped by small pairwise distances is therefore tautological with the categorical assignment. Please clarify the role of the distance in the grouping procedure.
  2. [Appendix B] The figure caption for Fig. 6 contains long runs of '/uni0000004b/uni00000044/...' escape sequences, making the caption unreadable. This is a compilation error that must be fixed.
  3. [Appendix C] Several datasets are described twice (DECIMER, SLAKE, NY CARTOON), with slightly different text. Please remove the duplicates.
  4. [Sec. 4.1.3] The description of the MixLoRA baseline states that all hyperparameters are preserved, but no details of the INSTA fine-tuning (architecture, epochs, LoRA rank, learning rate) are given. Please report these settings so the comparison is reproducible.
  5. [Table 1] Table 1 uses 'HCI' as a domain for ENRICO, which is not one of the four domains (healthcare, multimedia, affective computing, science) shown in Figure 2. Please make the domain taxonomy consistent.
  6. [Abstract and Sec. 4.2.2] The abstract claims 'new state-of-the-art results on all of them' on HEMM; however, Section 4.2.2 compares only to a small set of open-source models. Please temper the SoTA claim or provide a comparison to all HEMM-reported models.

Circularity Check

1 steps flagged · score 5.0 of 10

Partial circularity: Eq. (1) forces the Uniqueness labels for fixed-prompt image tasks, making part of the RUS grouping definitional rather than measured; the fine-tuning results themselves are not circular.

  1. self definitional [Sec. 3.1 Eq. (1); Sec. 3.3 categorization; App. C.2/C.7/C.8/C.12/C.18/C.20 prompts]
    "Uniqueness-dominant (GU): ∆1,2(Dk)≈1. Critical information is present in only one modality."

    Eq. (1) defines ∆ as average agreement between unimodal (f1,f2) and multimodal (fm) predictions. For the six datasets labeled Uniqueness, the text input is a fixed, task-independent prompt (e.g., App. C.12: 'Given a photo of a face, determine the facial expression. Choose from the following options: angry, disgust, fear, happy, neutral, sad, surprise.'); a text-only f2 has no task information, so δ(y2,ym)≈0, while a competent visual f1 gives δ(y1,ym)≈1, forcing ∆≈1. Sec. 3.3 then declares ∆≈1 'Uniqueness-dominant.' Sec. 4.1.1 presents the groups as obtained 'After computing the Multimodal Dataset Distance,' but for the entire Uniqueness group the label is forced by the fixed-prompt format, not by an empirical interaction property.

full rationale

Strictly speaking, the headline empirical claim is not circular: Table 2's accuracies are measured after fine-tuning, not derived from Eq. (1), and the baselines (single-task, unselective, INSTA, MixLoRA) are external comparisons. The self-citations to [60], [106], and [62] are to falsifiable, published work and do not by themselves force the result. However, one load-bearing component of the method does reduce by construction: the Uniqueness category. All six datasets in that group have a degenerate text modality (fixed prompt), so Eq. (1) mechanically gives ∆≈1, which Section 3.3 equates with uniqueness. The paper reports none of the ingredients needed to verify Eq. (1) (f1, f2, fm, δ, S, C, ∆ values), so the claimed computation is not distinguishable from hand-assigned labels for these tasks. This is partial circularity in the method's derivation chain, not in the empirical evaluation, so I score it 5 rather than 6-8. If the missing MI values were supplied and the fixed-prompt artifact corrected, the score would drop to 0-2.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new latent variables or mediators; the RUS categories are imported from prior information-theoretic work. The key unspoken cost is the set of unspecified choices in Eq. (1) that produce the grouping.

free parameters (2)
  • MI score categorization thresholds = approximately 2, 1, and 0
    Hand-set thresholds in Section 3.3 Step 1 define which MI scores count as redundancy, uniqueness, or synergy; no data-driven procedure or sensitivity analysis is given.
  • MI computation hyperparameters = unspecified (S, C, delta, unimodal and multimodal models)
    The MI score in Eq. (1) depends on S draws of C instances, a semantic similarity delta, and three models f1, f2, fm, none of which are specified in the paper, so the grouping is not reproducible.
assumptions (3)
  • domain assumption The MI score in Eq. (1) correctly captures the interaction type of a multimodal task.
    Section 3.1 asserts this on the basis of prior work [106, 60]; the paper provides no validation of the score against the assigned labels.
  • domain assumption For every selected dataset, a meaningful second modality exists.
    Tasks such as FER-2013 and RESISC45 are image classification tasks with a fixed text prompt; the text channel carries near-zero information, yet Eq. (1) requires a unimodal text model f2.
  • domain assumption The semantic similarity delta can be reliably computed for the open-ended outputs of these models.
    delta is defined abstractly in Section 3.1; no concrete implementation, for example a model-based similarity, is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping." pith.science (2026). https://pith.science/paper/XWO324FU

@misc{pith2026250602308,
  author       = {Pith},
  title        = {Pith review of: MINT: Multimodal Instruction Tuning with Multimodal Interaction Grouping},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/XWO324FU}},
  note         = {Machine review of arXiv:2506.02308}
}
read the original abstract

Recent advances in multimodal foundation models have achieved state-of-the-art performance across a range of tasks. These breakthroughs are largely driven by new pre-training paradigms that leverage large-scale, unlabeled multimodal data, followed by instruction fine-tuning on curated labeled datasets and high-quality prompts. While there is growing interest in scaling instruction fine-tuning to ever-larger datasets in both quantity and scale, our findings reveal that simply increasing the number of instruction-tuning tasks does not consistently yield better performance. Instead, we observe that grouping tasks by the common interactions across modalities, such as discovering redundant shared information, prioritizing modality selection with unique information, or requiring synergistic fusion to discover new information from both modalities, encourages the models to learn transferrable skills within a group while suppressing interference from mismatched tasks. To this end, we introduce MINT, a simple yet surprisingly effective task-grouping strategy based on the type of multimodal interaction. We demonstrate that the proposed method greatly outperforms existing task grouping baselines for multimodal instruction tuning, striking an effective balance between generalization and specialization.

Figures

Figures reproduced from arXiv: 2506.02308 by the authors.

Figure 1
Figure 1. Examples of multimodal instructions exhibiting redundancy, uniqueness, and synergy interactions, as [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our approach: For each multimodal instruction tuning dataset, spanning four domains: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Single-Task Fine-Tuning Comparison. We compare our MINT method against models that are fine-tuned individually on each dataset. • Single-Task Fine-Tuning: Fine-tuning the model on data from each individual dataset. • Unselective Multi-Task Fine-Tuning: Fine-tuning the model on the combined data from all tasks among all datasets, without any explicit grouping. • MINT: Fine-tuning the model separately on the data from… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Cross-Dataset Performance Comparison. The radar chart reports accuracies (%) on twelve benchmarks, where the outer rim represents the best score per dataset. The solid blue line is MINTThe dotted ˙ pink, cyan, and magenta curves (Similarity-G1 to Similarity-G3 fine-tun…
Figure 5
Figure 5. Figure 5: SoTA Open-source Models Comparison. Our method outperforms other open-source models, including the base model Qwen2-VL (7B) and the recent LLaMA-3.2-Vision (11B). Our results are obtained by fine-tuning Qwen2-VL on grouped datasets corresponding to different interactio…
Figure 6
Figure 6. Figure 6: Model performance comparison across datasets. “Single datasets fine-tuned” uses only in-domain data that exactly matches each test set, while “All datasets fine-tuned” mixes every dataset together without filtering. Our MINT method selectively aggregates related datase…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

141 extracted references · 35 canonical work pages

  1. [1]

    Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning.Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  2. [2]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: visual question answering. In2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 2425–2433. IEEE Computer Society,

    2015

  3. [3]

    Gated multimodal units for information fusion

    John Arevalo, Thamar Solorio, Manuel Montes-y Gómez, and Fabio A González. Gated multimodal units for information fusion. In5th International conference on learning representations 2017 workshop, 2017

    2017

  4. [4]

    Openflamingo: An open-source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models.arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  5. [5]

    Touchstone: Evaluating vision-language models by language models

    Shuai Bai, Shusheng Yang, Jinze Bai, Peng Wang, Xingxuan Zhang, Junyang Lin, Xinggang Wang, Chang Zhou, and Jingren Zhou. Touchstone: Evaluating vision-language models by language models. arXiv preprint arXiv:2308.16890, 2023

    arXiv 2023

  6. [6]

    Multimodal machine learning: A survey and taxonomy.IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018

    Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency. Multimodal machine learning: A survey and taxonomy.IEEE transactions on pattern analysis and machine intelligence, 41(2):423–443, 2018

    2018

  7. [7]

    Routledge, 2014

    John Bateman.Text and image: A critical introduction to the visual/verbal divide. Routledge, 2014

    2014

  8. [8]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Ta¸ sırlar. Introducing our multimodal models, 2023. URL https://www.adept.ai/blog/ fuyu-8b

    2023

Show all 141 references
  1. [9]

    Identifying beneficial task relations for multi-task learning in deep neural networks.arXiv preprint arXiv:1702.08303, 2017

    Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks.arXiv preprint arXiv:1702.08303, 2017

    2017 arXiv

  2. [10]

    Visit-bench: A benchmark for vision-language instruction following inspired by real-world use.arXiv preprint arXiv:2308.06595, 2023

    Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schimdt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use.arXiv preprint arXiv:2308.06595, 2023

    2023 arXiv

  3. [11]

    Decimer—hand- drawn molecule images dataset.Journal of Cheminformatics, 14(1):1–4, 2022

    Henning Otto Brinkhaus, Achim Zielesny, Christoph Steinbeck, and Kohulan Rajan. Decimer—hand- drawn molecule images dataset.Journal of Cheminformatics, 14(1):1–4, 2022

    2022

  4. [12]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models are few-shot learners. NeurIPS, 33:1877–1901, 2020

    1901

  5. [13]

    Multi-modal sarcasm detection in Twitter with hierarchical fusion model

    Yitao Cai, Huiyu Cai, and Xiaojun Wan. Multi-modal sarcasm detection in Twitter with hierarchical fusion model. In Anna Korhonen, David Traum, and Lluís Màrquez, editors,Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2506–2515, F...

    2019 doi

  6. [14]

    Multitask learning.Machine learning, 28:41–75, 1997

    Rich Caruana. Multitask learning.Machine learning, 28:41–75, 1997

    1997

  7. [15]

    Uniter: Universal image-text representation learning

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. InEuropean conference on computer vision, pages 104–120. Springer, 2020

    2020

  8. [16]

    Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art.Proceedings of the IEEE, 105(10):1865–1883, 2017

    2017

  9. [17]

    Scaling instruction-finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 10

    2024

  10. [18]

    Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.arXiv preprint arXiv:2311.03287, 2023

    Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges.arXiv preprint arXiv:2311.03287, 2023

    2023 arXiv

  11. [19]

    Climb: Data foundations for large scale multimodal clinical foundation models.arXiv preprint arXiv:2503.07667, 2025

    Wei Dai, Peilin Chen, Malinda Lu, Daniel Li, Haowen Wei, Hejie Cui, and Paul Pu Liang. Climb: Data foundations for large scale multimodal clinical foundation models.arXiv preprint arXiv:2503.07667, 2025

    2025 arXiv

  12. [20]

    Rico: A mobile app dataset for building data-driven design applications

    Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. Rico: A mobile app dataset for building data-driven design applications. In Proceedings of the 30th annual ACM symposium on user interface software and tech...

    2017

  13. [21]

    Multi-task learning for contextual bandits

    Aniket Anand Deshmukh, Urun Dogan, and Clay Scott. Multi-task learning for contextual bandits. Advances in neural information processing systems, 30, 2017

    2017

  14. [22]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. InNAACL-HLT (1), 2019

    2019

  15. [23]

    Regularized multi–task learning

    Theodoros Evgeniou and Massimiliano Pontil. Regularized multi–task learning. InProceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 109–117, 2004

    2004

  16. [24]

    A survey of current datasets for vision and language research

    Francis Ferraro, Nasrin Mostafazadeh, Ting-Hao Huang, Lucy Vanderwende, Jacob Devlin, Michel Galley, and Margaret Mitchell. A survey of current datasets for vision and language research. In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors,Proceedings of the 2015 Confe...

    2015 doi

  17. [25]

    Efficiently identifying task groupings for multi-task learning.Advances in Neural Information Processing Systems, 34:27503–27516, 2021

    Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning.Advances in Neural Information Processing Systems, 34:27503–27516, 2021

    2021

  18. [26]

    Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models.arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  19. [27]

    Enhancing knowledge transfer for task incremental learning with data-free subnetwork.Advances in Neural Information Processing Systems, 36: 68471–68484, 2023

    Qiang Gao, Xiaojun Shan, Yuchen Zhang, and Fan Zhou. Enhancing knowledge transfer for task incremental learning with data-free subnetwork.Advances in Neural Information Processing Systems, 36: 68471–68484, 2023

    2023

  20. [28]

    What makes the difference? an empirical comparison of fusion strategies for multimodal language analysis.Information Fusion, 66: 184–197, 2021

    Dimitris Gkoumas, Qiuchi Li, Christina Lioma, Yijun Yu, and Dawei Song. What makes the difference? an empirical comparison of fusion strategies for multimodal language analysis.Information Fusion, 66: 184–197, 2021

    2021

  21. [29]

    Challenges in representation learning: A report on three machine learning contests

    Ian J Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, et al. Challenges in representation learning: A report on three machine learning contests. InNeural Information Processing:...

    2013

  22. [30]

    Mixture of cluster-conditional lora experts for vision-language instruction tuning.arXiv preprint arXiv:2312.12379, 2023

    Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T Kwok, and Yu Zhang. Mixture of cluster-conditional lora experts for vision-language instruction tuning.arXiv preprint arXiv:2312.12379, 2023

    2023 arXiv

  23. [31]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  24. [32]

    Fusemoe: Mixture-of-experts transformers for fleximodal fusion.arXiv preprint arXiv:2402.03226, 2024

    Xing Han, Huy Nguyen, Carl Harris, Nhat Ho, and Suchi Saria. Fusemoe: Mixture-of-experts transformers for fleximodal fusion.arXiv preprint arXiv:2402.03226, 2024

    2024 arXiv

  25. [33]

    The movielens datasets: History and context.Acm transactions on interactive intelligent systems (tiis), 5(4):1–19, 2015

    F Maxwell Harper and Joseph A Konstan. The movielens datasets: History and context.Acm transactions on interactive intelligent systems (tiis), 5(4):1–19, 2015

    2015

  26. [34]

    Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering.arXiv preprint arXiv:2003.10286, 2020. 11

    2003 arXiv

  27. [35]

    understanding

    Jack Hessel, Ana Marasovi´c, Jena D Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. Do androids laugh at electric sheep? humor" understanding" benchmarks from the new yorker caption contest.ArXiv preprint, abs/2209.06293, 2022. URL https://arxiv.org...

    2022 arXiv

  28. [36]

    Framing image description as a ranking task: Data, models and evaluation metrics.Journal of Artificial Intelligence Research, 47:853–899, 2013

    Micah Hodosh, Peter Young, and Julia Hockenmaier. Framing image description as a ranking task: Data, models and evaluation metrics.Journal of Artificial Intelligence Research, 47:853–899, 2013

    2013

  29. [37]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  30. [38]

    Language is not all you need: Aligning perception with language models.Advances in Neural Information Processing Systems, 36, 2024

    Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, et al. Language is not all you need: Aligning perception with language models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  31. [39]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  32. [40]

    Memecap: A dataset for captioning and interpreting memes.arXiv preprint arXiv:2305.13703, 2023

    EunJeong Hwang and Vered Shwartz. Memecap: A dataset for captioning and interpreting memes.arXiv preprint arXiv:2305.13703, 2023

    2023 arXiv

  33. [41]

    Grounding, meaning and foundation models: Adventures in multimodal machine learning

    Douwe Kiela. Grounding, meaning and foundation models: Adventures in multimodal machine learning. InProceedings of the 30th ACM International Conference on Multimedia, pages 5–5, 2022

    2022

  34. [42]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611–2624, 2020

    2020

  35. [43]

    Grounding language models to images for multimodal inputs and outputs

    Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and outputs. 2023

    2023

  36. [44]

    Integrating text and image: Determining multimodal document intent in instagram posts

    Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, and Ajay Divakaran. Integrating text and image: Determining multimodal document intent in instagram posts. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Interna...

    2019

  37. [45]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    2018

  38. [46]

    Visual question answering in radiology (vqa-rad), Feb 2019

    Jason J Lau, Soumya Gayen, Dina Demner, and Asma Ben Abacha. Visual question answering in radiology (vqa-rad), Feb 2019. URLosf.io/89kps

    2019

  39. [47]

    Instruction matters: A simple yet effective task selection for optimized instruction tuning of specific tasks

    Changho Lee, Janghoon Han, Seonghyeon Ye, Stanley Jungkyu Choi, Honglak Lee, and Kyunghoon Bae. Instruction matters: A simple yet effective task selection for optimized instruction tuning of specific tasks. InProceedings of the 2024 Conference on Empirical Methods in Natural L...

    2024

  40. [48]

    INSTA: Instruction-based task selection for optimized instruction tuning.arXiv preprint arXiv:2404.16418, 2024

    Jinheon Lee, Jihyeon Kim, Cheoneum Park, and Se-Young Park. INSTA: Instruction-based task selection for optimized instruction tuning.arXiv preprint arXiv:2404.16418, 2024

    2024 arXiv

  41. [49]

    Holistic evaluation of text-to-image models.arXiv preprint arXiv:2311.04287, 2023

    Tony Lee, Michihiro Yasunaga, Chenlin Meng, Yifan Mai, Joon Sung Park, Agrim Gupta, Yunzhi Zhang, Deepak Narayanan, Hannah Benita Teufel, Marco Bellagente, et al. Holistic evaluation of text-to-image models.arXiv preprint arXiv:2311.04287, 2023

    2023 arXiv

  42. [50]

    Enrico: A dataset for topic modeling of mobile ui designs

    Luis A Leiva, Asutosh Hota, and Antti Oulasvirta. Enrico: A dataset for topic modeling of mobile ui designs. In22nd International Conference on Human-Computer Interaction with Mobile Devices and Services, pages 1–4, 2020

    2020

  43. [51]

    Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  44. [52]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational Conference on Machine Learning, pages 12888–12900. PMLR, 2022

    2022

  45. [53]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023. 12

    2023 arXiv

  46. [54]

    Visualbert: A simple and performant baseline for vision and language.arXiv preprint arXiv:1908.03557, 2019

    Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language.arXiv preprint arXiv:1908.03557, 2019

    1908 arXiv

  47. [55]

    Identifying task groupings for multi-task learning using pointwise v-usable information.arXiv preprint arXiv:2410.12774, 2024

    Yingya Li, Timothy Miller, Steven Bethard, and Guergana Savova. Identifying task groupings for multi-task learning using pointwise v-usable information.arXiv preprint arXiv:2410.12774, 2024

    2024 arXiv

  48. [56]

    Reform-eval: Evaluating large vision language models via unified re-formulation of task-oriented benchmarks.arXiv preprint arXiv:2310.02569, 2023

    Zejun Li, Ye Wang, Mengfei Du, Qingwen Liu, Binhao Wu, Jiwen Zhang, Chengxing Zhou, Zhihao Fan, Jie Fu, Jingjing Chen, et al. Reform-eval: Evaluating large vision language models via unified re-formulation of task-oriented benchmarks.arXiv preprint arXiv:2310.02569, 2023

    arXiv

  49. [57]

    Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models.arXiv preprint arXiv:2503.16843, 2025

    Jian Liang, Wenke Huang, Guancheng Wan, Qu Yang, and Mang Ye. Lorasculpt: Sculpting lora for harmonizing general and specialized knowledge in multimodal large language models.arXiv preprint arXiv:2503.16843, 2025

    2025 arXiv

  50. [58]

    Multibench: Multiscale benchmarks for multimodal representation learning

    Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Yufan Chen, Peter Wu, Michelle A Lee, Yuke Zhu, et al. Multibench: Multiscale benchmarks for multimodal representation learning. InThirty-fifth Conference on Neural Information Processing Systems Datas...

    2021

  51. [59]

    High-modality multimodal transformer: Quanti- fying modality & interaction heterogeneity for high-modality representation learning.arXiv preprint arXiv:2203.01311, 2022

    Paul Pu Liang, Yiwei Lyu, Xiang Fan, Jeffrey Tsaw, Yudong Liu, Shentong Mo, Dani Yogatama, Louis-Philippe Morency, and Ruslan Salakhutdinov. High-modality multimodal transformer: Quanti- fying modality & interaction heterogeneity for high-modality representation learning.arXiv...

    2022 arXiv

  52. [60]

    Quantifying & modeling multimodal interac- tions: An information decomposition framework

    Paul Pu Liang, Yun Cheng, Xiang Fan, Chun Kai Ling, Suzanne Nie, Richard J Chen, Zihao Deng, Nicholas Allen, Randy Auerbach, Faisal Mahmood, et al. Quantifying & modeling multimodal interac- tions: An information decomposition framework. InThirty-seventh Conference on Neural I...

    2023

  53. [61]

    Foundations & trends in multimodal machine learning: Principles, challenges, and open questions.ACM Computing Surveys, 2023

    Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. Foundations & trends in multimodal machine learning: Principles, challenges, and open questions.ACM Computing Surveys, 2023

    2023

  54. [62]

    Hemm: Holistic evaluation of multimodal foundation models

    Paul Pu Liang, Akshay Goindani, Talha Chafekar, Leena Mathur, Haofei Yu, Russ Salakhutdinov, and Louis-Philippe Morency. Hemm: Holistic evaluation of multimodal foundation models. InThe Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Tr...

    2024

  55. [63]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. InComputer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings,...

    2014

  56. [64]

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering. In2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  57. [65]

    Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  58. [66]

    What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.arXiv preprint arXiv:2312.15685, 2023

    Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning.arXiv preprint arXiv:2312.15685, 2023

    2023 arXiv

  59. [67]

    Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player?arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  60. [68]

    Nvila: Efficient frontier visual language models.arXiv preprint arXiv:2412.04468, 2024

    Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models.arXiv preprint arXiv:2412.04468, 2024

    2024 arXiv

  61. [69]

    The flan collection: Designing data and methods for effective instruction tuning

    Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. InInternational Conference on Machine Learning, pages 22631–22648. PMLR, 2023

    2023

  62. [70]

    Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.Advances in neural information processing systems, 32, 2019. 13

    2019

  63. [71]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. InThe 36th Conference on Neural Information Processing Systems (...

    2022

  64. [72]

    Modeling task relationships in multi-task learning with multi-gate mixture-of-experts

    Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. InProceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1930–1939, 2018

    1930

  65. [73]

    A taxonomy of relationships between images and text

    Emily E Marsh and Marilyn Domas White. A taxonomy of relationships between images and text. Journal of documentation, 59(6):647–672, 2003

    2003

  66. [74]

    Multi-task learning as a bargaining game.arXiv preprint arXiv:2202.01017, 2022

    Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game.arXiv preprint arXiv:2202.01017, 2022

    2022 arXiv

  67. [75]

    Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in neural information processing systems, 35:27730...

    2022

  68. [76]

    Kosmos- 2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos- 2: Grounding multimodal large language models to the world.arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  69. [77]

    To tune or not to tune? adapting pretrained representations to diverse tasks.arXiv preprint arXiv:1903.05987, 2019

    Matthew E Peters, Sebastian Ruder, and Noah A Smith. To tune or not to tune? adapting pretrained representations to diverse tasks.arXiv preprint arXiv:1903.05987, 2019

    1903 arXiv

  70. [78]

    Connecting vi- sion and language with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vi- sion and language with localized narratives. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 647–664....

    2020

  71. [79]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, and Sandhini Agarwal. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763. PMLR, 2021

    2021

  72. [80]

    Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model.Advances in Neural Information Processing Systems, 36:53728–53741, 2023

    2023

  73. [81]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020

    2020

  74. [82]

    Multitask prompted training enables zero-shot task generalization.arXiv preprint arXiv:2110.08207, 2021

    Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization.arXiv preprint arXiv:2110.08207, 2021

    arXiv

  75. [83]

    Chhavi Sharma, William Paka, Scott, Deepesh Bhageria, Amitava Das, Soujanya Poria, Tanmoy Chakraborty, and Björn Gambäck. Task Report: Memotion Analysis 1.0 @SemEval 2020: The Visuo- Lingual Metaphor! InProceedings of the 14th International Workshop on Semantic Evaluation (Sem...

    2020

  76. [84]

    Mome: Mixture of multimodal experts for generalist multimodal large language models.arXiv preprint arXiv:2407.12709, 2024

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for generalist multimodal large language models.arXiv preprint arXiv:2407.12709, 2024

    2024 arXiv

  77. [85]

    Multimodal instruction tuning with conditional mixture of lora.arXiv preprint arXiv:2402.15896, 2024

    Ying Shen, Zhiyang Xu, Qifan Wang, Yu Cheng, Wenpeng Yin, and Lifu Huang. Multimodal instruction tuning with conditional mixture of lora.arXiv preprint arXiv:2402.15896, 2024

    2024 arXiv

  78. [86]

    A principled approach for learning task similarity in multitask learning.arXiv preprint arXiv:1903.09109, 2019

    Changjian Shui, Mahdieh Abbasi, Louis-Émile Robitaille, Boyu Wang, and Christian Gagné. A principled approach for learning task similarity in multitask learning.arXiv preprint arXiv:1903.09109, 2019

    1903 arXiv

  79. [87]

    Guibas, Jitendra Malik, and Silvio Savarese

    Trevor Standley, Amir Roshan Zamir, Dawn Chen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? InProceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Ev...

    2020

  80. [88]

    Vl-bert: Pre-training of generic visual-linguistic representations.arXiv preprint arXiv:1908.08530, 2019

    Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations.arXiv preprint arXiv:1908.08530, 2019. 14

    1908 arXiv

  81. [89]

    A corpus of natural language for visual reasoning

    Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217–223, 2017

    2017

  82. [90]

    Multimodal transformer for unaligned multimodal language sequences

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. Multimodal transformer for unaligned multimodal language sequences. InProceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pag...

    2019

  83. [91]

    Multimodal few-shot learning with frozen language models.Advances in Neural Information Processing Systems, 34: 200–212, 2021

    Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models.Advances in Neural Information Processing Systems, 34: 200–212, 2021

    2021

  84. [92]

    The inaturalist species classification and detection dataset-supplementary material.Reptilia, 32(400):1–3, 2017

    Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset-supplementary material.Reptilia, 32(400):1–3, 2017

    2017

  85. [93]

    Git: A generative image-to-text transformer for vision and language.arXiv preprint arXiv:2205.14100, 2022

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language.arXiv preprint arXiv:2205.14100, 2022

    2022 arXiv

  86. [94]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  87. [95]

    Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.arXiv preprint arXiv:2204.07705, 2022

    Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super- naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks.arXiv p...

    2022 arXiv

  88. [96]

    Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners.arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  89. [97]

    On the road with gpt-4v(ision): Early explorations of visual-language model on autonomous driving, 2023

    Licheng Wen, Xuemeng Yang, Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, Zheng Zhu, Shaoyan Sun, Yeqi Bai, Xinyu Cai, Min Dou, Shuanglu Hu, and Botian Shi. On the road with gpt-4v(ision): Early explorations of visual-language mo...

    2023

  90. [98]

    Nonnegative decomposition of multivariate information.arXiv preprint arXiv:1004.2515, 2010

    Paul L Williams and Randall D Beer. Nonnegative decomposition of multivariate information.arXiv preprint arXiv:1004.2515, 2010

    2010 arXiv

  91. [99]

    Multimodal dataset distillation for image-text retrieval

    Xindi Wu, Zhiwei Deng, and Olga Russakovsky. Multimodal dataset distillation for image-text retrieval. arXiv preprint arXiv:2308.07545, 2023

    2023 arXiv

  92. [100]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.arXiv preprint arXiv:2306.09265, 2023

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.arXiv preprint arXiv:2306.09265, 2023

    2023 arXiv

  93. [101]

    Multimodal chatgpt for medical applications: an experimental study of gpt-4v.arXiv preprint arXiv:2310.19061, 2023

    Zhiling Yan, Kai Zhang, Rong Zhou, Lifang He, Xiang Li, and Lichao Sun. Multimodal chatgpt for medical applications: an experimental study of gpt-4v.arXiv preprint arXiv:2310.19061, 2023

    2023 arXiv

  94. [102]

    Bag-of-visual-words and spatial extensions for land-use classification

    Yi Yang and Shawn Newsam. Bag-of-visual-words and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL international conference on advances in geographic information systems, pages 270–279, 2010

    2010

  95. [103]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  96. [104]

    Irfl: Image recognition of figurative language.ArXiv preprint, abs/2303.15445, 2023

    Ron Yosef, Yonatan Bitton, and Dafna Shahaf. Irfl: Image recognition of figurative language.ArXiv preprint, abs/2303.15445, 2023. URLhttps://arxiv.org/abs/2303.15445

    2023 arXiv

  97. [105]

    From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.TACL, 2:67–78, 2014

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions.TACL, 2:67–78, 2014

    2014

  98. [106]

    Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts

    Haofei Yu, Zhengyang Qi, Lawrence Jang, Russ Salakhutdinov, Louis-Philippe Morency, and Paul Pu Liang. Mmoe: Enhancing multimodal models with mixtures of multimodal interaction experts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, ...

    2024

  99. [107]

    Gradient surgery for multi-task learning.Advances in neural information processing systems, 33:5824–5836, 2020

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning.Advances in neural information processing systems, 33:5824–5836, 2020

    2020

  100. [108]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  101. [109]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProceedings of the IEEE/CVF Conference on Co...

    2024

  102. [110]

    Magicbrush: A manually annotated dataset for instruction-guided image editing

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing. InAdvances in Neural Information Processing Systems, 2023

    2023

  103. [111]

    Transfer adaptation learning: A decade survey.IEEE Transactions on Neural Networks and Learning Systems, 35(1):23–44, 2022

    Lei Zhang and Xinbo Gao. Transfer adaptation learning: A decade survey.IEEE Transactions on Neural Networks and Learning Systems, 35(1):23–44, 2022

    2022

  104. [112]

    Semantic understanding of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset.International Journal of Computer Vision, 127:302–321, 2019

    2019

  105. [113]

    Single datasets fine-tuned

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment.Advances in Neural Information Processing Systems, 36:55006–55021, 2023. 16 A MINT details We fine-tune our MINT based ...

    2023

  106. [115]

    yes" or

    VQAis a benchmark dataset comprising pairs of images and corresponding free-form, open-ended questions. Answering these questions often requires fine-grained recognition of objects and activities within the image, and in some cases, commonsense reasoning. A significant portion...

  107. [117]

    The SMILES format encodes the three-dimensional molecular structure into a linear string of symbols

    DECIMERis a dataset of hand-drawn molecular structure images, each paired with its corresponding SMILES (Simplified Molecular Input Line Entry System) representation. The SMILES format encodes the three-dimensional molecular structure into a linear string of symbols. Solving t...

  108. [118]

    Each question requires the model to select the correct answer from a given set of options

    SCIENCEQAis a multiple-choice question dataset covering diverse science domains, including natural science, social science, and language science. Each question requires the model to select the correct answer from a given set of options. Supplementary materials such as lecture ...

  109. [119]

    Annotations have been done by experienced physicians and a medical knowl- edge base for medical visual question answering

    SLAKEis a medical visual question-answering dataset that consists of image and question- answer pairs. Annotations have been done by experienced physicians and a medical knowl- edge base for medical visual question answering. The dataset consists of Yes/No type of questions as...

  110. [120]

    Annotations were provided by experienced physicians and curated using a medical knowledge base, ensuring domain-specific accuracy

    SLAKEis a medical visual question answering (VQA) dataset comprising image-question- answer triplets. Annotations were provided by experienced physicians and curated using a medical knowledge base, ensuring domain-specific accuracy. The questions include both binary (Yes/No) t...

  111. [121]

    The images were manually extracted from the USGS National Map Urban Area Imagery, covering various urban regions across the United States

    UCMERCED LAND USEis a dataset for land use classification, comprising aerial images categorized into 21 distinct classes. The images were manually extracted from the USGS National Map Urban Area Imagery, covering various urban regions across the United States. All possible cla...

  112. [122]

    It builds upon the RICO dataset [20], with additional human annotations rating each UI design as either good or bad

    ENRICOis a topic classification dataset for mobile user interface (UI) screenshots. It builds upon the RICO dataset [20], with additional human annotations rating each UI design as either good or bad. Each sample is associated with a UI class—such as calculator, camera, chat, ...

  113. [123]

    Each movie can belong to multiple genres

    MM-IMDBis a genre classification dataset consisting of movie posters and corresponding plot descriptions. Each movie can belong to multiple genres. The dataset was constructed using the MovieLens 20M dataset [33], from which metadata such as genre, plot, release year, and othe...

  114. [124]

    The images are sourced from MedPix 1, an open-access radiology image database

    VQARADis a visual question answering dataset based on radiology images. The images are sourced from MedPix 1, an open-access radiology image database. The dataset was manually curated by clinical annotators, including medical students and senior radiologists. 1https://medpix.n...

  115. [125]

    It follows similar data collection and annotation protocols, providing a rich set of images with corresponding human-written captions

    FLICKR30Kis an image captioning dataset sourced from Flickr 2, extending the dataset introduced by [36]. It follows similar data collection and annotation protocols, providing a rich set of images with corresponding human-written captions. Split:Evaluation is conducted on the ...

  116. [126]

    The images were collected using the Google Search API, and OpenCV was employed to detect and extract face bounding boxes

    FER-2013is a widely used dataset for facial expression recognition, where each facial image is classified into one of seven emotion categories. The images were collected using the Google Search API, and OpenCV was employed to detect and extract face bounding boxes. Split:Evalu...

  117. [127]

    The dataset is formulated based on taking in the image and caption to predict how funny the pair is based on the normalized number of votes

    NY CARTOONis collected from the weekly New Yorker magazine cartoon captioning contest 3, where readers are tasked to give a humorous caption for a cartoon image and the funniest captions are selected based on public votes. The dataset is formulated based on taking in the image...

  118. [128]

    not funny,

    NY CARTOONis a dataset derived from the weekly cartoon captioning contest hosted by *The New Yorker*4, where readers submit humorous captions for cartoon images, and the most amusing entries are selected based on public votes. Each data sample pairs a cartoon with a caption, a...

  119. [129]

    Each sample includes an image and corresponding textual instructions, enabling guided image manipulation

    MAGICBRUSHis an instruction-based image editing dataset featuring both single-turn and multi-turn editing tasks. Each sample includes an image and corresponding textual instructions, enabling guided image manipulation. The images are sampled from the MS COCO dataset [63], and ...

  120. [130]

    Captions were generated through a two-round annotation process by human workers on Amazon Mechanical Turk

    MEMECAPis a meme captioning dataset, with images sourced from the r/memes subreddit 7. Captions were generated through a two-round annotation process by human workers on Amazon Mechanical Turk. For evaluation, the model is provided with the title and a description of the meme ...

  121. [131]

    yes" or

    HATEFULMEMESis a multimodal classification dataset released as part of a challenge hosted by Meta, designed to assess whether a meme image paired with its textual caption expresses hateful intent. The images were sourced from Getty Images8 and annotated by a third-party platfo...

  122. [132]

    The images and labels are sourced from the iNaturalist platform9

    INATURALISTis a large-scale image classification dataset encompassing over 5,000 wildlife species of plants and animals. The images and labels are sourced from the iNaturalist platform9. The task involves identifying the species depicted in a given image. Unlike other classifi...

    2021

  123. [133]

    true” or “false

    NLVR(Natural Language for Visual Reasoning) is a visual reasoning dataset consisting of image-text pairs. Each image is synthetically generated by randomly sampling objects and their properties. Crowdworkers are then asked to write natural language sentences describing these i...

  124. [134]

    The images were collected from Google Earth by experts in remote sensing image interpretation

    RESISC45is a remote sensing image classification dataset containing aerial land use scenes categorized into 45 distinct classes. The images were collected from Google Earth by experts in remote sensing image interpretation. During evaluation, all class names are included in th...

  125. [135]

    We use the COCO subset of this dataset for the task of image generation, where the model generates an image based on a textual description

    Localized Narratives (COCO subset)(LNCOCO) is derived from the Localized Narratives dataset [78], which augments images from COCO [63], Flickr30k [105], and ADE20K [112] with detailed spatial annotations and spoken captions. We use the COCO subset of this dataset for the task ...

    2017

  126. [136]

    The Q-former uses an attention mechanism to fuse image queries with input text, producing a joint representation that is passed to the decoder for response generation

    BLIP-2is a vision-language model that combines a pre-trained image encoder and a pre- trained large language model (LLM) via a lightweight Q-former module. The Q-former uses an attention mechanism to fuse image queries with input text, producing a joint representation that is ...

  127. [137]

    Unlike models limited to single-image inputs (e.g., BLIP2 or MiniGPT-4), OPENFLAMINGOsupports multi- image processing within a single sample

    OPENFLAMINGOis an open-source implementation of the Flamingo model [ 1], designed for multimodal reasoning over sequences of interleaved images and texts. Unlike models limited to single-image inputs (e.g., BLIP2 or MiniGPT-4), OPENFLAMINGOsupports multi- image processing with...

  128. [138]

    Its architecture is identical to that of Persimmon-8B12, and thus inherits its model categorization

    FUYU-8Bis a decoder-only transformer model that processes visual inputs by linearly projecting image patches into the first layer of the decoder. Its architecture is identical to that of Persimmon-8B12, and thus inherits its model categorization. Persimmon-8B contains 9.3 bill...

  129. [139]

    It is trained using the next-token prediction objective

    KOSMOS-2is a causal Transformer-based language model, extending the architecture of Kosmos-1 [38]. It is trained using the next-token prediction objective. In addition to the original pre-training data used in Kosmos-1, KOSMOS-2 incorporates grounded image-text pairs for impro...

  130. [140]

    A vision foundation model encodes the input image, and a visual abstractor module summarizes the visual features

    MPLUG-OWLis a multimodal model that integrates visual and textual inputs using a staged fusion strategy. A vision foundation model encodes the input image, and a visual abstractor module summarizes the visual features. These abstracted representations are then combined with te...

  131. [141]

    The resulting visual tokens are interleaved with textual inputs and passed to a pre-trained Qwen2 language model, which serves as the decoder backbone

    Qwen2-VLadopts a two-tower architecture, in which a ViT-based image encoder extracts visual features that are then aligned with the language space via a lightweight projection module. The resulting visual tokens are interleaved with textual inputs and passed to a pre-trained Q...

  132. [142]

    A vision encoder—typically a CLIP ViT model—extracts dense image embeddings, which are projected and injected into the language model through cross- attention layers

    LLaMA3.2-Vextends the LLaMA 3.2 language model with vision capabilities via a gated multimodal adapter. A vision encoder—typically a CLIP ViT model—extracts dense image embeddings, which are projected and injected into the language model through cross- attention layers. Compar...

  133. [2015]

    URLhttps://doi.org/10.1109/ICCV.2015.279

    doi: 10.1109/ICCV .2015.279. URLhttps://doi.org/10.1109/ICCV.2015.279

    2015 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.