Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Current audio-visual LLMs score near chance when inputs are mismatched or missing a modality, and a proposed preference-training method lifts accuracy by up to 30.19% across nine tasks.

desk verdict Useful benchmark, fragile training claims; the zero-shot evaluation deserves peer review, but CAVPref's headline gains need far more scrutiny before being taken at face value. read the letter →

arxiv 2501.02135 v1 pith:U3FOD42Y submitted 2025-01-03 cs.CV cs.AI

Sanjoy Chowdhury , Sayan Nag , Subhrajyoti Dasgupta , Yaoting Wang , Mohamed Elhoseiny , Ruohan Gao , Dinesh Manocha This is my paper · ORCID
classification cs.CVcs.AI
keywords audio-visuallargelanguagemodelstrustworthinessbenchmarkadversarialattackscompositionalreasoningmodalitydependencypreferenceoptimizationdistributionallyrobustmultiple-choiceQA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

AVTrustBench is a 600,000-sample multiple-choice benchmark that probes audio-visual large language models under three conditions previous benchmarks did not cover: adversarial inputs where the answer is missing or irrelevant or a modality is swapped, compositional inputs where the order or attribute binding of events is changed, and inputs where one modality is deleted. The paper evaluates 13 models and finds that on the adversarial and compositional suites most of them perform near chance, and that all models are better at noticing a bad video than a bad audio track. The authors then propose CAVPref, a model-agnostic training loss built from three preference terms -- text, video, and audio -- each wrapped in a distributionally robust average, and report gains up to 30.19% over zero-shot accuracy across all nine tasks. A sympathetic reader would take away that trust in audio-visual LLMs should not be assumed out of the box, and that a training signal explicitly conditioned on both modalities measurably improves it.

What carries the argument

The load-bearing mechanism is a calibrated preference-optimization objective with three additive terms: $L_y$ for text preference between a winning and losing response, $L_V$ for visual conditioning (the winning response under the correct versus a mismatched video), and $L_A$ for audio conditioning (the winning response under the correct versus a mismatched audio). Each term is scaled by a $\beta$ that grows with the normalized similarity difference between the winning and losing scenarios, using CLAP score differences for $L_y$ and an audio-visual similarity metric for $L_V$ and $L_A$. Every term is then passed through a distributionally robust transform $-\lambda \log \mathbb{E}_P[e^{L/\lambda}]$, which replaces the average loss with a worst-case expected loss over distributions that stay close to the training data. The final loss is $L_{\mathrm{CAVPref}} = L_y^R + \eta L_V^R + \gamma L_A^R$, where the binary switches $\eta$ and $\gamma$ turn on only for tasks in which the video or audio stream, respectively, is the perturbed one. This structure is what lets the model learn to flag inconsistencies in each modality rather than merely imitate text preferences.

What would settle it

Take 2,000 held-out AVTrustBench questions that were not part of the 500-per-axis validation set and have a small human panel answer them independently; if human-model agreement drops well below the claimed Spearman 0.979, or if humans cannot reach near the 91% ceiling on the full test set, the benchmark's scores would not measure what the paper says. A second check: for a random subset of MVT and MAT questions, have annotators judge whether both modalities are genuinely required; if many are answerable from one modality alone, the modality-dependency results collapse.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central claim is that current audio-visual large language models do not genuinely understand audio-visual consistency. When offered multiple-choice answers with no correct option, when the video or audio is swapped for an unrelated event, when two events are reordered, or when the same words are bound to different attributes, most of the 13 evaluated models pick an offered option instead of saying that the input is inconsistent, and their accuracy hovers around chance on the adversarial and compositional suites. The paper further claims that this failure is fixable by training: CAVPref, which adds visual- and audio-conditioned preference terms to a standard DPO loss and applies a distributionally robust transform, improves accuracy by up to 30.19 percentage points across all nine tasks, with the largest gains on exactly the tasks that require detecting a missing or mismatched modality.

Load-bearing premise

The benchmark's scores stand on the assumption that the semi-automatically generated answer keys are correct for the roughly 99 percent of samples not checked by humans; if the look-up tables, LLM-generated distractors, or the single-modality filtering introduced errors, every accuracy number shifts.

Editorial extensions

If this is right

  • If the benchmark scores are right, no current audio-visual LLM is reliable enough to deploy in settings where a wrong answer can cause harm; near-chance accuracy on adversarial and compositional tasks means the models are effectively guessing.
  • The up-to-30.19% gains on three open models suggest a model-agnostic training recipe that can be applied to any audio-visual LLM without changing its architecture.
  • The large gap between video-removal and audio-removal accuracy implies that existing models lean on video; CAVPref narrows that gap, so modality-balanced training is an achievable goal.
  • The compositional suite's near-chance performance, together with the paper's 7B-versus-13B comparison, implies that simply scaling up the language backbone will not fix compositional understanding.
  • The release of the 600K-sample suite provides a standardized testbed that future audio-visual LLMs can be measured against for trustworthiness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the same three-term construction could be applied to image-only or audio-only LLMs by dropping the unused modality term, suggesting a general recipe for modality-consistency fine-tuning that the paper does not state.
  • Editorial inference: because the evaluation adopts circular assessment (a question is solved only if answered correctly under all rotations of the answer choices), the reported accuracies are lower-bound estimates; real open-ended reliability may be even worse than the tables show.
  • Editorial inference: a testable extension would be to measure per-category accuracy before and after CAVPref on a held-out long-tail audio-visual set, checking whether the robustness module transfers beyond the categories in this benchmark.
  • Editorial inference: another testable extension is to regenerate the preference data with a different large language model and rerun CAVPref, which would separate the method's contribution from the particular choice of distractor generation.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces AVTrustBench, a 600K-sample multiple-choice benchmark for evaluating audio-visual LLMs (AVLLMs) across three axes: adversarial attack (MCIT, ICIT, MVIT, MAIT), compositional reasoning (COT-Stitch, COT-Swap, CAT), and modality-specific dependency (MVT, MAT). The authors report zero-shot evaluations of 13 AVLLMs, finding that most perform near or only slightly above chance. To improve robustness, they propose CAVPref, a preference optimization method with modality-conditioned losses and a distributionally robust optimization (DRO) component, and report gains up to 30.19% across all nine tasks on three AVLLMs (and seven more in the supplement). The paper also includes a human study estimating 91.27% accuracy on a selected subset.

Significance. If the benchmark labels are reliable, AVTrustBench fills a real gap: no existing benchmark systematically tests audio-visual trustworthiness, and the reported failure modes are important. The CAVPref method is a sensible extension of DPO that conditions on audio and visual correctness and uses DRO to mitigate tail-category underperformance; the broader applicability is evidenced by transfer results to VideoBench and MVBench (Supp. D.6) for one model. The paper provides the community with a large, multi-task evaluation suite and a clear mitigation strategy. The main risks are the limited validation of the semi-automatic ground-truth generation and the tuning of CAVPref hyperparameters on the benchmark itself.

major comments (4)
  1. [Sec. 3.2 / Supp. N.1] The benchmark's core claims rest on the correctness of the semi-automatic QA labels. The validation reported in Supp. N.1 samples only 500 examples per axis (0.25% of the 600K corpus) and reports a Spearman correlation of 0.979, which is not a per-item label accuracy. Because reported accuracies are 20–50%, even a small fraction of erroneous labels can change model rankings and the model-human gap. I request a per-item human accuracy on a random sample of the AVTrustBench-test set, stratified by the nine tasks, plus agreement rates for each construction step (look-up table replacement, LLM distractor generation, modality-dependency filtering).
  2. [Supp. A.1] Modality-specific dependency tasks (MVT, MAT) assume that both audio and video are required to answer the question. The pipeline removes ~50% of MUSIC-AVQA samples for this suite via a manual 'careful inspection' (Supp. A.1), yet no inter-annotator agreement or audit is reported. Residual single-modality questions would make the expected answer ('the video is missing' / 'the audio is missing') ambiguous, rendering the MVT/MAT scores uninterpretable. Please quantify the reliability of the exclusion filter, e.g., by having independent annotators re-label a random sample of the retained and excluded questions.
  3. [Sec. 5.3 / Eq. (5)] The hyperparameters λy, λV, λA are chosen by ablation on MCIT, MVIT, and MAIT from the same benchmark used to report the headline improvement (Fig. 6). The paper does not specify whether these tasks overlap with the evaluation split. If the same tasks (or a subset) are used for both hyperparameter selection and final reporting, the 30.19% gain is optimistically biased. Please use a separate validation split for hyperparameter selection, or show that the performance gains persist across a range of λ values (e.g., λ ∈ {0.2, 0.5, 1.0, 2.0}) on all nine tasks.
  4. [Supp. N.1] The human study in Supp. N.1 manually selects 50 successful and 50 failed GPT-4o cases per task, yielding a 91.27% average accuracy. This sample is deliberately biased and cannot estimate the actual human accuracy on the full benchmark; the claim that tasks are 'straightforward for human subjects' is therefore not established by this protocol. Please provide a human evaluation on a random sample (e.g., 50–100 per task) drawn from the full test set, or at least report accuracy separately for the two halves and justify the selection.
minor comments (5)
  1. [Sec. 3.2] The abstract says '600K samples' but the test set is 181K; please clarify the train/validation/test split and the source of training samples for CAVPref.
  2. [Table 1] The 'Temporal?' column uses checkmarks for several benchmarks, but the meaning is not always clear (e.g., does 'Temporal' mean the tasks require temporal reasoning or that the benchmark uses videos?); consider a footnote.
  3. [Eqs. (4)–(5)] Eq. (4) introduces a general D_f divergence, but the closed-form solution in Eq. (5) is derived for KL divergence (Supp. D.2); please unify the notation or state when D_f is restricted to KL.
  4. [References] The reference list contains duplicate entries ([35] and [36] are the same Silkie paper); please consolidate.
  5. [Figure 2] The right panel's dimension-wise averages would be easier to read if the three dimensions were labeled on the bars or in the caption.

Circularity Check

2 steps flagged · score 4.0 of 10

CAVPref's headline gain is partly test-set-tuned and partly anchored in a self-cited AV similarity metric; the zero-shot AVLLM evaluation itself is independent.

  1. fitted input called prediction [Sec. 5.3 Ablations and Sec. 3.2 AVTrustBench Statistics]
    "For all our evaluations we use the AVTRUST BENCH -test set comprising 181K samples. ... We systematically ablate the values of λy, λV and λA in the Eq. 5 and assess the performance on MCIT, MVIT and MAIT tasks respectively (Fig. 6). We observe that a value of 1.0 is the best for both λy and λV whereas for λA the best performance was obtained for a value of 0.8."

    The CAVPref improvements reported in Tab. 5, including the headline 'up to 30.19%', are computed on the same AVTrustBench tasks and the same 181K-sample test set on which the λ hyperparameters were selected by ablation. No separate validation split is described for the λ search. Consequently, the reported gain is a post-hoc, test-set-tuned quantity rather than an out-of-sample prediction: the hyperparameters were fit to the very tasks whose improvement is claimed. The external transfer results on VideoBench and MVBench (Supp D.6) provide partial independent support, but the headline number itself remains selected on the evaluation target.

  2. self citation load bearing [Sec. 5.1 CAVPref, Eqs. (2)-(3) and β definition]
    "For automatic selection of β, we propose β as an increasing function of (batch) normalized similarity score difference ∆S between winning and losing scenarios: β = g(∆S) = 0.9∆S + 0.1. For βy (Eq. 1), we use CLAP score differences, and for βV and βA (Eqs. 2 - 3), we use A V Similarity Metric (A VSM) [11, 12] differences as∆S."

    The β schedule that controls the CAVPref preference-optimization update is computed from AVSM, a metric imported from the authors' own prior papers [11,12] (Chowdhury et al., ECCV and CVPR 2024). The present paper does not define or independently validate AVSM; it cites the authors' previous work as the source of the metric. To the degree that the claimed improvement depends on the β values, the training signal is anchored in a self-cited construction rather than in a derivation given here. This is partial circularity: the zero-shot benchmark evaluation is independent, but the mitigation method's internal weighting is not self-contained.

full rationale

The central diagnostic claims of AVTrustBench — that 13 audio-visual LLMs perform poorly on adversarial, compositional, and modality-dependency tasks — are zero-shot measurements on a newly constructed benchmark and do not reduce to the paper's own equations or to its citations; those results are independent evidence. The 0.979 Spearman correlation on 500 samples per axis is a small validation sample, but that is a data-quality/coverage limitation, not circularity. The circular content is concentrated in the CAVPref improvement claim. First, the λ hyperparameters are selected by ablating performance on MCIT, MVIT, and MAIT, which are exactly among the nine tasks for which 'up to 30.19%' gains are then reported, and the paper states that all evaluations use the same 181K-sample test set; this is a fitted-input-called-prediction pattern. Second, the β weighting in the training objective is computed using AVSM from the authors' prior work [11,12], making the method partially self-referential. These issues do not force the zero-shot model rankings or the brittleness conclusions, and the external VideoBench/MVBench transfer results in Supp D.6 provide some independent support for the training strategy. The score is therefore 4: some self-citation and test-set-dependent tuning, while the benchmark's main evaluation content remains independent.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claims rest on the validity of the automatically generated benchmark labels and on the fairness of the CAVPref evaluation. Free parameters include the DRO regularization lambdas selected by ablation on the benchmark and the hand-picked beta calibration constants. The invented entity list is empty; CAVPref is a training objective, not a new physical or conceptual entity. The axioms capture the generalization of human validation and the correctness of the look-up tables, which are load-bearing for the benchmark's ground truth.

free parameters (3)
  • lambda_y, lambda_V, lambda_A (DRO regularization) = 1.0, 1.0, 0.8
    Selected via ablation on MCIT, MVIT, MAIT tasks in Sec. 5.3 / Fig. 6; these values are used in the reported CAVPref results.
  • beta calibration constants (0.9, 0.1) = beta = 0.9 * DeltaS + 0.1
    Hand-picked linear calibration in Sec. 5.1; no sensitivity analysis beyond lambda.
  • Tsim threshold for AudioSet filtering = 0.70
    In Supplementary A, samples with Tsim < 0.70 are excluded from compositional reasoning data; this threshold is chosen by hand and affects the compositionality task distribution.
assumptions (4)
  • domain assumption Human validation on 1500 samples (500 per axis) generalizes to the full 600K benchmark.
    Supplementary N and B.5; the Spearman 0.979 on 500 samples per axis is used to claim benchmark validity.
  • domain assumption Look-up tables T1/T2/T3 correctly encode unrelated audio-visual categories and answer sets for adversarial tasks.
    Supplementary A.2; MCIT/ICIT/MVIT/MAIT correctness depends on these manually constructed mappings.
  • domain assumption GPT-4 choice extraction and CircularEval yield a fair correctness measure.
    Supplementary B.2/B.3; used for all reported accuracies.
  • standard math DRO closed-form L = -lambda log E_P[exp(L/lambda)] is valid as derived.
    Supplementary D.2; the derivation is sketchy but the result is standard Duchi-Namkoong DRO.

how reviews work

0 comments
Cite this review

Pith. "Pith review of AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs." pith.science (2026). https://pith.science/paper/U3FOD42Y

@misc{pith2026250102135,
  author       = {Pith},
  title        = {Pith review of: AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U3FOD42Y}},
  note         = {Machine review of arXiv:2501.02135}
}
read the original abstract

With the rapid advancement of Multi-modal Large Language Models (MLLMs), several diagnostic benchmarks have recently been developed to assess these models' multi-modal reasoning proficiency. However, these benchmarks are restricted to assessing primarily the visual aspect and do not examine the holistic audio-visual (AV) understanding. Moreover, currently, there are no benchmarks that investigate the capabilities of AVLLMs to calibrate their responses when presented with perturbed inputs. To this end, we introduce Audio-Visual Trustworthiness assessment Benchmark (AVTrustBench), comprising 600K samples spanning over 9 meticulously crafted tasks, evaluating the capabilities of AVLLMs across three distinct dimensions: Adversarial attack, Compositional reasoning, and Modality-specific dependency. Using our benchmark we extensively evaluate 13 state-of-the-art AVLLMs. The findings reveal that the majority of existing models fall significantly short of achieving human-like comprehension, offering valuable insights for future research directions. To alleviate the limitations in the existing approaches, we further propose a robust, model-agnostic calibrated audio-visual preference optimization based training strategy CAVPref, obtaining a gain up to 30.19% across all 9 tasks. We will publicly release our code and benchmark to facilitate future research in this direction.

Figures

Figures reproduced from arXiv: 2501.02135 by the authors.

Figure 1
Figure 1. Introducing AVTRUSTBENCH and CAVPref. We present AVTRUSTBENCH, a new benchmark comprising three challenging yet unexplored axes, i.e., Adversarial Attack , Compositional Reasoning , and Modality Dependency , and evaluate SOTA Audio-Visual LLMs (AVLLMs) on this benchmark. We observe that these models demonstrate poor performances under these settings. To alleviate these limitations, we propose a novel AVLLM-agnostic … view at source ↗
Figure 2
Figure 2. AVTRUSTBENCH statistics and AVLLMs leaderboard. (Left) Task-wise data distribution. Our benchmark comprises 9 diverse tasks spanning over 3 dimensions. (Right) Performance comparison on AVTRUSTBENCH. Values represent dimension-wise averages. 2. Related Work Building Multi-modal LLMs. Inspired by the success of large language models [10, 49, 63], recent work has ex￾panded LLMs to multi-modal understanding, leveraging… view at source ↗
Figure 3
Figure 3. Task definitions: AVTRUSTBENCH comprises a total of 9 tasks tasks MCIT, ICIT, MVIT and MAIT from Adversarial attack, COT-Stitch, COT-Swap and CAT from Compositional reasoning and MAT and MVT from Modality-specific dependency respectively. The goal of each dimension is to critically assess the robustness of existing AVLLMs under different modes of challenges. In each case, the AVLLMs are presented with a multiple-cho… view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Qualitative results. We report top 8 models’ performance on three representative tasks MCIT, COT-Swap and MAT. GPT-4o consistently outperforms open-source models. Under instruction setting we append the phrase “If the correct answer is not present respond with None of …
Figure 5
Figure 5. Figure 5: Overview of CAVPref. We formulate a distributionally robust AV preferential optimization objective to incorporate the multi-modal relationships across different modalities and counter the tailing effect across diverse categories in the dataset. suggesting that a bigger…
Figure 6
Figure 6. Figure 6: Performance changes with varying values of λy, λV and λA on MCIT, MVIT and MAIT tasks respectively. have improved substantially; (iii) The significance of AV conditioning over DPO is particularly evident in tasks like MVIT, MAIT, COT-Swap, MVT, and MAT, where DPO shows…
Figure 7
Figure 7. Figure 7: Leaderboards for zero-shot evaluation on 9 different tasks in AVT [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]
Figure 8
Figure 8. Figure 8: Distribution of different question categories across AVQA and MUSIC-AVQA datasets. [PITH_FULL_IMAGE:figures/full_fig_p018_8.png]
Figure 9
Figure 9. Figure 9: Performance comparison of all open source models on MCIT task under ZS, DPO and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p021_9.png]
Figure 10
Figure 10. Figure 10: Performance comparison of all open source models on ICIT task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p022_10.png]
Figure 11
Figure 11. Figure 11: Performance comparison of all open source models on MVIT task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p022_11.png]
Figure 12
Figure 12. Figure 12: Performance comparison of all open source models on MAIT task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p027_12.png]
Figure 13
Figure 13. Figure 13: Performance comparison of all open source models on COT-Stitch task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p028_13.png]
Figure 14
Figure 14. Figure 14: Performance comparison of all open source models on COT-Swap task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p028_14.png]
Figure 15
Figure 15. Figure 15: Performance comparison of all open source models on CAT task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p029_15.png]
Figure 16
Figure 16. Figure 16: Performance comparison of all open source models on MVT task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p029_16.png]
Figure 17
Figure 17. Figure 17: Performance comparison of all open source models on MAT task under ZS, DPO, and CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p030_17.png]
Figure 18
Figure 18. Figure 18: Example scenario depicting that most AVLLMs struggle in Common Sense Reasoning. [PITH_FULL_IMAGE:figures/full_fig_p030_18.png]
Figure 19
Figure 19. Figure 19: Failure cases of video-SALMONN, Video-LLaMA2, and Bay-CAT after training with CAVPref. [PITH_FULL_IMAGE:figures/full_fig_p031_19.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Bridging Ears and Eyes: Analyzing Audio and Visual Large Language Models to Humans in Visible Sound Recognition and Reducing Their Sensory Gap via Cross-Modal Distillation

    cs.SD 2025-05 conditional novelty 7.0 of 10

    Audio and visual LLMs show a sensory gap that resembles the human ears-versus-eyes gap, and cross-modal distillation with a learned routing switch raises a visual-model-trained audio LLM's test accuracy from 69% to 89%.

  2. FutureOmni: Evaluating Future Forecasting from Omni-Modal Context for Multimodal LLMs

    cs.CL 2026-01 conditional novelty 6.0 of 10

    FutureOmni, a 919-video, 1,034-question audio-visual future-forecasting benchmark, shows top MLLMs reach only 64.8% accuracy, and OFF tuning improves open models.

  3. MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks

    cs.CV 2025-06 conditional novelty 6.0 of 10

    AVHaystacks is a new 3100-question benchmark for audio-visual QA across 500 videos, and the MAGNET multi-agent pipeline beats current baselines on it.

  4. EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perception

    cs.CV 2025-06 conditional novelty 5.0 of 10

    A joint distillation and policy-learning framework claims near-teacher accuracy on egocentric action recognition, active speaker localization, and behavior anticipation at a fraction of the compute.

Reference graph

Works this paper leans on

91 extracted references · 18 canonical work pages · cited by 4 Pith papers

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems , 35:23716–23736, 2022

    2022

  3. [3]

    Openflamingo: An open- source framework for training large autoregressive vision- language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open- source framework for training large autoregressive vision- language models. arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  4. [4]

    Activitynet: A large-scale video bench- mark for human activity understanding

    Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video bench- mark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961–970, 2015

    2015

  5. [5]

    X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160, 2023

    arXiv 2023

  6. [6]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023

    arXiv

  7. [7]

    Shikra: Unleashing multimodal llm’s referential dialogue magic

    Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal llm’s referential dialogue magic. arXiv preprint arXiv:2306.15195, 2023

    arXiv 2023

  8. [8]

    Vast: A vision-audio-subtitle- text omni-modality foundation model and dataset

    Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: A vision-audio-subtitle- text omni-modality foundation model and dataset. Advances in Neural Information Processing Systems, 36, 2024

    2024

Show all 91 references
  1. [9]

    Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024

    2024 arXiv

  2. [10]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, 2023

    2023

  3. [11]

    Meerkat: Audio-visual large language model for grounding in space and time

    Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In European Conference on Computer Vision, 2024

    2024

  4. [12]

    Melfusion: Synthesizing music from image and language cues using diffusion models

    Sanjoy Chowdhury, Sayan Nag, KJ Joseph, Balaji Vasan Srinivasan, and Dinesh Manocha. Melfusion: Synthesizing music from image and language cues using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26826–26835, 2024

    2024

  5. [13]

    Scaling instruction- finetuned language models

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024

    2024

  6. [14]

    Think you have solved question answering? try arc, the ai2 reason- ing challenge

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reason- ing challenge. arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  7. [15]

    Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning, 2023

    2023

  8. [16]

    Enhancing large vision language models with self-training on image comprehension

    Yihe Deng, Pan Lu, Fan Yin, Ziniu Hu, Sheng Shen, James Zou, Kai-Wei Chang, and Wei Wang. Enhancing large vision language models with self-training on image comprehension. arXiv preprint arXiv:2405.19716, 2024

    2024 arXiv

  9. [17]

    Learning models with uniform performance via distributionally robust opti- mization

    John C Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust opti- mization. The Annals of Statistics, 49(3):1378–1406, 2021

    2021

  10. [18]

    Clap learning audio concepts from natural language supervision

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2023

    2023

  11. [19]

    Audio set: An ontology and human-labeled dataset for audio events

    Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international confer- ence on acoustics, speech and signal processing...

    2017

  12. [20]

    Imagebind: One embedding space to bind them all

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15180–15190, 2023

    2023

  13. [21]

    Onellm: One framework to align all modalities with language

    Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. arXiv preprint arXiv:2312.03700, 2023

    2023 arXiv

  14. [22]

    Imagebind-llm: Multi-modality instruction tuning

    Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint arXiv:2309.03905, 2023

    2023 arXiv

  15. [23]

    Measuring massive multitask language understanding

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Man- tas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  16. [24]

    Dogs’ responses to visual, auditory, and olfactory cat-related cues

    Christy L Hoffman, Miranda K Workman, Natalie Roberts, and Stephanie Handley. Dogs’ responses to visual, auditory, and olfactory cat-related cues. Applied Animal Behaviour Science, 188:50–58, 2017

    2017

  17. [25]

    Perceiver: General per- ception with iterative attention

    Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General per- ception with iterative attention. In International conference on machine learning, pages 4651–4664. PMLR, 2021

    2021

  18. [26]

    Hallucination augmented contrastive learning for multimodal large language model

    Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

    2024

  19. [27]

    Fgaif: Aligning large vision- language models with fine-grained ai feedback.arXiv preprint arXiv:2404.05046, 2024

    Liqiang Jing and Xinya Du. Fgaif: Aligning large vision- language models with fine-grained ai feedback.arXiv preprint arXiv:2404.05046, 2024

    2024 arXiv

  20. [28]

    Tvqa: Localized, compositional video question answering

    Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696, 2018

    2018 arXiv

  21. [29]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  22. [30]

    Otter: A multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023

    2023 arXiv

  23. [31]

    Learning to answer questions in dynamic audio-visual scenarios

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19108–19118, 2022

    2022

  24. [32]

    Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip- 2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023

    2023 arXiv

  25. [33]

    Videochat: Chat-centric video understanding

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355, 2023

    2023 arXiv

  26. [34]

    Mvbench: A comprehensive multi-modal video understand- ing benchmark

    Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understand- ing benchmark. arXiv preprint arXiv:2311.17005, 2023

    2023 arXiv

  27. [36]

    Silkie: Preference distillation for large visual language models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.10665, 2023

    2023 arXiv

  28. [37]

    Evaluating object hallucina- tion in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models. arXiv preprint arXiv:2305.10355, 2023

    2023 arXiv

  29. [38]

    Truthfulqa: Measuring how models mimic human falsehoods

    Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021

    2021 arXiv

  30. [39]

    Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models

    Fuxiao Liu, Tianrui Guan, Zongxia Li, Lichang Chen, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusionbench: You see what you think? or you think what you see? an image-context reasoning benchmark challenging for gpt-4v (ision), llava-1.5, and other multi-modality models...

    arXiv

  31. [40]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024

    2024

  32. [41]

    Statistical rejection sampling improves preference optimization

    Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mo- hammad Saleh, Peter J Liu, and Jialu Liu. Statistical rejection sampling improves preference optimization. 2024

    2024

  33. [42]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  34. [43]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023

    2023 arXiv

  35. [44]

    Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093, 2023

    2023 arXiv

  36. [45]

    Video-chatgpt: Towards detailed video understanding via large vision and language models

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424, 2023

    2023 arXiv

  37. [46]

    Audio-visual generalised zero-shot learning with cross-modal attention and language

    Otniel-Bogdan Mercea, Lukas Riesch, A Koepke, and Zeynep Akata. Audio-visual generalised zero-shot learning with cross-modal attention and language. In Proceedings of the IEEE/CVF conference on computer vision and pattern recog- nition, pages 10553–10563, 2022

    2022

  38. [47]

    Video-bench: A comprehen- sive benchmark and toolkit for evaluating video-based large language models

    Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehen- sive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103, 2023

    2023 arXiv

  39. [48]

    Hello gpt-4, 2024

    OpenAI. Hello gpt-4, 2024

    2024

  40. [49]

    Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:27730–27744, 2022

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Car- roll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback.Advances in Neural Information Processing Systems, 35:277...

    2022

  41. [50]

    X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning

    Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction-aware representations to llms and emergent cross-modal reasoning. arXiv prepr...

    2023 arXiv

  42. [51]

    Coordinated joint multimodal embeddings for gen- eralized audio-visual zero-shot classification and retrieval of videos

    Kranti Parida, Neeraj Matiyali, Tanaya Guha, and Gaurav Sharma. Coordinated joint multimodal embeddings for gen- eralized audio-visual zero-shot classification and retrieval of videos. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 3...

    2020

  43. [52]

    Kosmos-2: Ground- ing multimodal large language models to the world

    Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Ground- ing multimodal large language models to the world. arXiv preprint arXiv:2306.14824, 2023

    2023 arXiv

  44. [53]

    Strengthening multimodal large language model with bootstrapped preference optimiza- tion

    Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimiza- tion. arXiv preprint arXiv:2403.08730, 2024

    2024 arXiv

  45. [54]

    Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision-language model

    Shraman Pramanick, Guangxing Han, Rui Hou, Sayan Nag, Ser-Nam Lim, Nicolas Ballas, Qifan Wang, Rama Chellappa, and Amjad Almahairi. Jack of all tasks, master of many: Designing general-purpose coarse-to-fine vision-language model. arXiv preprint arXiv:2312.12423, 2023

    2023 arXiv

  46. [55]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  47. [56]

    Direct prefer- ence optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct prefer- ence optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  48. [57]

    Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context

    Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrit- twieser, et al. Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. arXiv...

    2024 arXiv

  49. [58]

    Timechat: A time-sensitive multimodal large lan- guage model for long video understanding

    Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large lan- guage model for long video understanding. arXiv preprint arXiv:2312.02051, 2023

    2023 arXiv

  50. [59]

    Mitigating object halluci- nation via data augmented contrastive tuning

    Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan Ö Arık, and Tomas Pfister. Mitigating object halluci- nation via data augmented contrastive tuning. arXiv preprint arXiv:2405.18654, 2024

    2024 arXiv

  51. [60]

    Pandagpt: One model to instruction-follow them all

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355, 2023

    2023 arXiv

  52. [61]

    video-salmonn: Speech-enhanced audio-visual large language models

    Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, Yuxuan Wang, and Chao Zhang. video-salmonn: Speech-enhanced audio-visual large language models. arXiv preprint arXiv:2406.15704, 2024

    2024 arXiv

  53. [62]

    Aligning large multi- modal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chun- yuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu- Xiong Wang, Yiming Yang, et al. Aligning large multi- modal models with factually augmented rlhf. arXiv preprint arXiv:2309.14525, 2023

    2023 arXiv

  54. [63]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu- lab/ stanford_alpaca, 2023

    2023

  55. [64]

    Movieqa: Understanding stories in movies through question-answering

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Anto- nio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4631–4640, 2016

    2016

  56. [65]

    Winoground: Probing vision and language models for visio- linguistic compositionality

    Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kiela, and Candace Ross. Winoground: Probing vision and language models for visio- linguistic compositionality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition ...

    2022

  57. [66]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Mar- tinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Roz- ière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  58. [67]

    Llama 2: Open foundation and fine-tuned chat models, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Am- jad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https://arxiv. org/abs/2307.09288, 2023

    2023 arXiv

  59. [68]

    What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223, 2023

    Guangzhi Wang, Yixiao Ge, Xiaohan Ding, Mohan Kankan- halli, and Ying Shan. What makes for good visual tokenizers for large language models? arXiv preprint arXiv:2305.12223, 2023

    2023 arXiv

  60. [69]

    Vision transformer with deformable attention

    Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4794–4803, 2022

    2022

  61. [70]

    De- tecting and mitigating hallucination in large vision lan- guage models via fine-grained ai feedback

    Wenyi Xiao, Ziwei Huang, Leilei Gan, Wanggui He, Haoyuan Li, Zhelun Yu, Hao Jiang, Fei Wu, and Linchao Zhu. De- tecting and mitigating hallucination in large vision lan- guage models via fine-grained ai feedback. arXiv preprint arXiv:2404.14233, 2024

    2024 arXiv

  62. [71]

    Funqa: Towards surprising video comprehension

    Binzhu Xie, Sicheng Zhang, Zitang Zhou, Bo Li, Yuanhan Zhang, Jack Hessel, Jingkang Yang, and Ziwei Liu. Funqa: Towards surprising video comprehension. arXiv preprint arXiv:2306.14899, 2023

    2023 arXiv

  63. [72]

    Msr-vtt: A large video description dataset for bridging video and language

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016

    2016

  64. [73]

    Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models

    Peng Xu, Wenqi Shao, Kaipeng Zhang, Peng Gao, Shuo Liu, Meng Lei, Fanqing Meng, Siyuan Huang, Yu Qiao, and Ping Luo. Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models. arXiv preprint arXiv:2306.09265, 2023

    2023 arXiv

  65. [74]

    Avqa: A dataset for audio- visual question answering on videos

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio- visual question answering on videos. In Proceedings of the 30th ACM International Conference on Multimedia , pages 3480–3491, 2022

    2022

  66. [75]

    mplug-owl: Modularization empowers large language models with multimodality

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023

    2023 arXiv

  67. [76]

    Cat: enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios

    Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, and Xiaochun Cao. Cat: enhancing multimodal large language model to answer questions in dynamic audio-visual scenarios. arXiv preprint arXiv:2403.04640, 2024

    2024 arXiv

  68. [77]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023

    2023 arXiv

  69. [78]

    Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark

    Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingn- ing Liu, Mukai Li, Xiaoshui Huang, Zhiyong Wang, Lu Sheng, Lei Bai, et al. Lamm: Language-assisted multi-modal instruction-tuning dataset, framework, and benchmark. Ad- vances in Neural Information Processing Systems, 36, 2024

    2024

  70. [79]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Co...

    2024

  71. [80]

    Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness

    Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning mllms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint arXiv:2405.17220, 2024

    2024

  72. [81]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  73. [82]

    Florence: A new foundation model for computer vision

    Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021

    2021 arXiv

  74. [83]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  75. [84]

    Video-llama: An instruction-tuned audio-visual language model for video un- derstanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video un- derstanding. arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  76. [85]

    Llama- adapter: Efficient fine-tuning of language models with zero- init attention

    Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama- adapter: Efficient fine-tuning of language models with zero- init attention. arXiv preprint arXiv:2303.16199, 2023

    2023 arXiv

  77. [86]

    Gpt4roi: Instruction tuning large language model on region-of-interest

    Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest. arXiv preprint arXiv:2307.03601, 2023

    2023 arXiv

  78. [87]

    Chatbridge: Bridging modalities with large language model as a language catalyst

    Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst. arXiv preprint arXiv:2305.16103, 2023

    2023 arXiv

  79. [88]

    Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimiza- tion

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimiza- tion. arXiv preprint arXiv:2311.16839, 2023

    2023 arXiv

  80. [89]

    Aligning modalities in vision large lan- guage models via preference fine-tuning

    Yiyang Zhou, Chenhang Cui, Rafael Rafailov, Chelsea Finn, and Huaxiu Yao. Aligning modalities in vision large lan- guage models via preference fine-tuning. arXiv preprint arXiv:2402.11411, 2024

    2024 arXiv

  81. [90]

    None of the above

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. AVTRUST BENCH : Assessing and Enhancing Reliability and Robustness in Audio-Vis...

    2023 arXiv

  82. [91]

    If step 1 fails, we provide GPT-4 with the question, choices, and model prediction

    establish the effectiveness of GPT-4 as a choice extrac- tor. If step 1 fails, we provide GPT-4 with the question, choices, and model prediction. and instruct it to align the prediction with one of the given choices and produce the label. If there is no match found, GPT-4 retu...

  83. [92]

    None of the above

    we feed each question N times to the A VLLMs where N is the number of choices by making a circular shift to the choices. We attribute the A VLLM to successfully solving a question if it correctly predicts the answer in all circular passes. Once an A VLLM fails in any of the pa...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.