Pith. sign in

REVIEW 4 major objections 5 minor 71 references

SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SilVar is an open-source multimodal model that lets users ask an image questions by voice and get reasoned answers with bounding boxes, at a measurable accuracy cost relative to text.

desk verdict A genuinely useful speech-reasoning dataset wrapped in a paper whose SOTA claim is refuted by its own tables. read the letter →

arxiv 2412.16771 v1 pith:6ZVNCA6Q submitted 2024-12-21 cs.CV

Tan-Hanh Pham , Hoang-Nam Le , Phu-Vinh Nguyen , Chris Ngo , Truong-Son Hy This is my paper · ORCID
classification cs.CV
keywords speech-drivenvisualquestionansweringobjectlocalizationmultimodalreasoningWhisperaudioencoderCLIPvisionLLaMA3.1speechinstructiontuningSilVar-Benchdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SilVar is an open-source multimodal model that takes a spoken or typed question together with an image and produces a reasoned text answer plus bounding boxes for the object in question. The paper is trying to establish that speech can be a first-class instruction modality for visual question answering and object localization, rather than just a front-end transcription step, and it contributes a two-stage training pipeline plus a new benchmark to make that case. The pipeline aligns a Whisper audio encoder, a CLIP visual encoder, and the LLaMA 3.1-8B language model, and the new dataset supplies 998 COCO images with GPT-4-generated conversational, simple-reasoning, and complex-reasoning speech questions. The abstract states that SilVar achieves state-of-the-art results on MMMU and ScienceQA, while the paper's own tables show a more modest outcome: SilVar is competitive with strong text-based models but trails them on both benchmarks. Read in good faith, the contribution is a working open-source speech-driven reasoning pipeline and evidence that spoken instructions cost a measurable but not prohibitive amount of accuracy.

What carries the argument

The carrying mechanism is a two-stage speech-instruction tuning pipeline built from three open-source components: Whisper (an audio encoder), CLIP (a visual encoder), and LLaMA 3.1-8B (the language backbone). Stage 1 trains the audio side to align speech with text in the reasoning domain using science and multimodal-reasoning datasets; Stage 2 fine-tunes the whole model to answer from direct audio input on those datasets plus the new SilVar-Bench. The adapter design is intentionally simple, with linear or MLP projections for audio and two linear layers with GELU for vision, and the ablation shows that larger Transformer audio adapters buy almost nothing, implying that the Whisper encoder's final layer already carries the needed structure. The new dataset supplies the speech-instruction signal: 998 COCO images, three reasoning question types per object, human-verified answers, manual bounding boxes, and synthetic speech from over fifty voices.

What would settle it

Inspect the released data splits and training scripts: if any of the 775 preprocessed MMMU validation samples appear in Stage 1 or Stage 2 training, the reported 31.8 validation score is in-distribution and cannot be compared with published MMMU results; a clean check is to run the released model on the official MMMU test set and compare it with the same baselines.

Watch

Extended reading notes

Core claim

The central claim, on the paper's own terms, is that a single open-source model can align audio, image, and language features well enough to answer complex reasoning questions and localize objects from spoken instructions. SilVar does this by encoding speech with Whisper into 768-dimensional features, projecting them to the LLaMA input space, and concatenating them with CLIP visual tokens before the language model generates text and bounding boxes. The paper's experiments indicate that complex-reasoning speech prompts outperform simple and conversational speech prompts on its own benchmark, and that end-to-end training with speech slightly improves the MMMU and ScienceQA scores over the pipeline variant. The same tables show that text instructions consistently score higher than speech instructions, so the defensible finding is that speech-driven reasoning works at a measurable accuracy cost, not that it surpasses text-based state of the art.

Load-bearing premise

The whole benchmark comparison rests on the assumption that the 775 MMMU validation samples used in the speech pipeline were not included in either training stage, but the paper never states that exclusion.

Editorial extensions

If this is right

  • Voice becomes a usable input modality for open-source visual reasoning, not just a speech-to-text front end, so applications can be built without proprietary speech models.
  • Spoken complex-reasoning prompts produce better grounded answers than simple or conversational prompts, so prompt-design techniques transfer to the audio channel.
  • A model trained on SilVar-Bench can output bounding boxes from speech, opening a route to hands-free assistive tools such as scene description and navigation aids.
  • Because the paper's own measurements show text instructions are more accurate, deployed speech-driven systems should keep a text fallback for high-stakes questions.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial: the abstract's 'state-of-the-art' phrasing is stronger than the tables support; the reproducible contribution is the first open speech-driven reasoning pipeline and benchmark, not a new accuracy record.
  • Editorial: the speech-to-text gap is likely dominated by the tiny 39-million-parameter Whisper encoder; retraining with a larger ASR encoder and measuring the same benchmarks would test this directly.
  • Editorial: the MMMU comparison is only meaningful if the 775 preprocessed validation samples were excluded from both training stages, which the paper does not state; releasing the exact split would settle the question.
  • Editorial: because the benchmark questions are GPT-4-generated from captions, some questions may leak the target object through wording; a no-image human baseline would quantify how much of the 'reasoning' is linguistic rather than visual.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. SilVar is an end-to-end speech-driven visual language model built on CLIP, Whisper, and LLaMA 3.1-8B, trained in two stages on ScienceQA, MMMU, LISA, and a newly introduced SilVar dataset for speech-based reasoning and object localization. The paper claims state-of-the-art performance on MMMU and ScienceQA while also investigating conversational, simple, and complex reasoning instructions in both speech and text modalities. The model architecture, training pipeline, dataset generation procedure, and a small ablation of audio adapters are described.

Significance. Speech-driven reasoning for visual question answering is a timely and useful direction, and the release of a new dataset with human-verified bounding boxes and speech instructions is a positive contribution. The two-stage training pipeline and the comparison of linear, MLP, and Transformer audio adapters are also useful engineering results. However, the central claim of state-of-the-art performance is contradicted by the paper's own tables, and the MMMU evaluation is compromised by the likely overlap between validation samples used in training and the reported validation evaluation. The dataset and code release, if made accessible with a proper URL, would be a valuable resource for the community even if the benchmark claims are revised downward.

major comments (4)
  1. [§5.2, Table 4 and §5.3, Table 5] The abstract and conclusion claim that SilVar 'achieves SOTA performance on the MMMU and ScienceQA benchmarks,' but the paper's own tables contradict this. On MMMU validation, SilVar Text scores 31.8, below LLaVA-1.5-13B (36.4) and Qwen-VL-7B-Chat (35.9). On ScienceQA, SilVar Speech scores 63.21, far below LLaVA-13B (90.92), LaVIN-13B (90.83), and Chat-UniVi (88.78). The paper does not define a restricted comparison class such as 'speech-driven open-source models,' and Table 5 contains no speech-input baselines at all. The SOTA claim is therefore not supported by the reported results.
  2. [§3, Table 1, Table 2, §5.2] The MMMU evaluation is likely in-distribution. The paper reports curating a subset that includes 775 samples from the MMMU validation set for preprocessing, Table 2 lists MMMU as used in both Stage 1 and Stage 2 training, and Section 5.2 evaluates only on the MMMU validation set. The paper never states that the 775 preprocessed validation samples were excluded from training. If they were not excluded, the reported validation score of 31.8 is an in-distribution accuracy, not a legitimate benchmark result. The authors must clarify the exact data split and, ideally, evaluate on the official MMMU test set.
  3. [§5.2] The justification for evaluating only on the validation set—'SOTA models achieve similar performance on both test and validation datasets'—is not a substitute for reporting test-set numbers and does not support a SOTA claim. The official MMMU test set contains 10,500 samples, and the validation set is not a held-out benchmark if any part of it contributed to training. The paper should report official test-set accuracy or clearly and verifiably document that the validation samples used in preprocessing were fully excluded from all training stages.
  4. [§5.1, Table 3] The localization accuracy at IoU = 0.5 is between 21.11% and 26.32% across all instruction types. Since object localization is one of the paper's two core contributions, these numbers need to be placed in context with baselines or an explicit discussion of what accuracy level constitutes a meaningful capability. Without any comparison for the localization task, the reported numbers do not substantiate the claim that SilVar can perform reasoning-based object localization effectively.
minor comments (5)
  1. [§5.3] 'ScieneQA' is a typo that should be 'ScienceQA'.
  2. [Throughout] Cross-references are often given as bare numbers (e.g., 'as shown in 2', 'as shown in 1') instead of 'Figure 2' and 'Figure 1'; the captions should also number all figures consistently.
  3. [Table 4] The 'Test' column heading is misleading because SilVar rows report no test values; consider renaming the column to 'Val/Test' or clarifying which models have official test scores.
  4. [Abstract and §7] The abstract and conclusion state 'Our code and dataset are available here,' but no URL, repository identifier, or anonymous link is provided in the manuscript.
  5. [Table 3] The caption refers to 'blue' and 'cyan' highlights, which are not reproducible in monochrome print; please use boldface or symbols instead.

Circularity Check

1 steps flagged · score 6.0 of 10

MMMU benchmark result is in-distribution because the paper's own pipeline trains on samples curated from the MMMU validation set and then reports validation-only scores as benchmark predictions.

  1. fitted input called prediction [Section 3 (GPT-assisted Data Generation), Table 2 (Datasets and stages), Section 4.2 (Training), Section 5.2 (MMMU-bench)]
    "we curate a subset consisting of 132 samples from the training set and 775 samples from the validation set. ... we train the model on the ScienceQA and MMMU datasets using speech instructions ... since the test dataset is significantly larger than the validation dataset, and SOTA models achieve similar performance on both test and validation datasets, we intentionally evaluate our model only on the validation set. ... Dataset Data type Stage 1 Stage 2 ... MMMU Text, image x x"

    The 775 MMMU samples drawn from the validation set are processed into the speech-instruction dataset, and Table 2 marks MMMU as training data in both Stage 1 and Stage 2. Table 4 then reports validation-set accuracy (31.8 text, 30.2 speech, 30.4 end-to-end) as the benchmark result. If any of those 775 validation-derived samples are in the training mixture, the reported scores are in-distribution fits rather than held-out predictions. The paper never states that the curated validation subset was excluded from training, and it does not evaluate on the official MMMU test set, so the benchmark 'prediction' is not independent of its training input by construction.

full rationale

The only concrete circular step is the MMMU evaluation. The paper curates 775 samples from the MMMU validation set (Section 3), marks MMMU as training data in both stages (Table 2 and Section 4.2), and then reports validation-only scores in Table 4, justifying this by saying the test set is larger. Unless those 775 validation-derived samples were explicitly held out of Stage 1 and Stage 2 training, the reported 31.8/30.2/30.4 are partially fitted rather than independent benchmark predictions. No exclusion statement appears in the paper. The ScienceQA evaluation, by contrast, is a standard held-out benchmark report against text baselines; it is weak evidence for the 'SOTA' claim but not circular. The abstract's 'SOTA' claim is additionally contradicted by the paper's own Tables 4 and 5 (e.g., 31.8 vs 36.4 for LLaVA-1.5-13B on MMMU val; 63.21 vs 90.92 for LLaVA-13B on ScienceQA), but that is a correctness/falsification problem, not circularity. No load-bearing self-citation, uniqueness import, or ansatz-smuggling chain appears; the architecture uses standard external checkpoints (CLIP, Whisper, LLaMA-3.1-8B), and the held-out portions of the new SilVar dataset provide independent content. The central circularity is therefore confined to the MMMU benchmark result, which reduces to the training set by construction if the curated validation samples were used as described.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its central claims rest on the faithfulness of pretrained encoders, the quality of GPT-4 generated data, the validity of MMMU validation as an evaluation set, and the representativeness of TTS voices. The MMMU validation assumption is the most fragile because it may make the reported benchmark score circular.

free parameters (3)
  • MMMU training/validation subset sizes = 132 train, 775 validation samples
    The paper arbitrarily curates subsets of MMMU for speech preprocessing; the choice directly determines which samples are used for training and evaluation, and is not justified.
  • Learning rate and schedule = 1e-5 with warmup 1e-5 and cosine decay
    Chosen by hand for training; the reported results depend on these values, but no sensitivity analysis is given.
  • Audio max length = 1500
    Ad hoc truncation length for Whisper features; may cut longer instructions and affect reasoning accuracy.
assumptions (4)
  • domain assumption Whisper tiny encoder outputs retain enough semantic information for reasoning-level VQA after finetuning.
    The whole pipeline hinges on speech features being aligned to text meaning, as stated in Section 4.1.
  • domain assumption GPT-4-generated questions and answers verified by humans are correct and cover the intended reasoning.
    The SilVar dataset is created with GPT-4 and manual verification; no inter-annotator reliability is reported (Section 3).
  • ad hoc to paper The MMMU validation set can be used as a benchmark after the paper preprocesses and trains on MMMU subsets that include validation samples.
    Section 3 uses 775 validation samples; Section 5.2 evaluates on validation only and never on the test set, without proving the validation samples were held out.
  • domain assumption TTS voices from Google Cloud are representative of natural speech for evaluating speech-instruction VQA.
    The paper converts all text to speech with over 50 synthesized voices (Section 3); no robustness analysis to natural speech is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization." pith.science (2026). https://pith.science/paper/6ZVNCA6Q

@misc{pith2026241216771,
  author       = {Pith},
  title        = {Pith review of: SilVar: Speech Driven Multimodal Model for Reasoning Visual Question Answering and Object Localization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ZVNCA6Q}},
  note         = {Machine review of arXiv:2412.16771}
}
read the original abstract

Visual Language Models have demonstrated remarkable capabilities across tasks, including visual question answering and image captioning. However, most models rely on text-based instructions, limiting their effectiveness in human-machine interactions. Moreover, the quality of language models depends on reasoning and prompting techniques, such as COT, which remain underexplored when using speech instructions. To address these challenges, we propose SilVar, a novel end-to-end multimodal model that uses speech instructions for reasoning in visual question answering. In addition, we investigate reasoning techniques with levels including conversational, simple, and complex speech instruction. SilVar is built upon CLIP, Whisper, and LLaMA 3.1-8B, enabling intuitive interactions by allowing users to provide verbal or text instructions. To this end, we introduce a dataset designed to challenge models with speech-based reasoning tasks for object localization. This dataset enhances the model ability to process and explain visual scenes from spoken input, moving beyond object recognition to reasoning-based interactions. The experiments show that SilVar achieves SOTA performance on the MMMU and ScienceQA benchmarks despite the challenge of speech-based instructions. We believe SilVar will inspire next-generation multimodal reasoning models, toward expert artificial general intelligence. Our code and dataset are available here.

Figures

Figures reproduced from arXiv: 2412.16771 by the authors.

Figure 1
Figure 1. An example from our SilVar-bench dataset, focusing on reasoning speech instructions with different types: con [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the SilVar model architecture, integrating visual and audio instruction for reasoning text generation [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

71 extracted references · 39 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    The sound of healthcare: Improving medical transcrip- tion asr accuracy with large language models

    Ayo Adedeji, Sarita Joshi, and Brendan Doohan. The sound of healthcare: Improving medical transcrip- tion asr accuracy with large language models. arXiv preprint arXiv:2402.07658, 2024

    arXiv 2024

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022

    2022

  4. [4]

    Vqa: Visual question answering

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425–2433, 2015

    work page 2015

  5. [5]

    Speecht5: Unified-modal encoder- decoder pre-training for spoken language processing

    Junyi Ao, Rui Wang, Long Zhou, Chengyi Wang, Shuo Ren, Yu Wu, Shujie Liu, Tom Ko, Qing Li, Yu Zhang, et al. Speecht5: Unified-modal encoder- decoder pre-training for spoken language processing. arXiv preprint arXiv:2110.07205, 2021

    arXiv

  6. [6]

    Openflamingo: An open-source framework for train- ing large autoregressive vision-language models

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hes- sel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, 10 Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for train- ing large autoregressive vision-language models. arXiv preprint arXiv:2308.01390, 2023

    arXiv 2023

  7. [7]

    wav2vec 2.0: A framework for self-supervised learning of speech representations

    Alexei Baevski, Yuhao Zhou, Abdelrahman Mo- hamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems , 33:12449–12460, 2020

    work page 2020

  8. [8]

    Qwen-vl: A frontier large vision- language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint arXiv:2308.12966, 2023

    arXiv 2023

Show all 71 references
  1. [9]

    Introducing our multimodal models, 2023

    Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sa˘gnak Tas ¸ırlar. Introducing our multimodal models, 2023

    2023

  2. [10]

    Language models are few-shot learn- ers

    Tom B Brown. Language models are few-shot learn- ers. arXiv preprint arXiv:2005.14165, 2020

    2005 arXiv

  3. [11]

    Minigpt-v2: large language model as a unified interface for vision-language multi-task learning

    Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mo- hamed Elhoseiny. Minigpt-v2: large language model as a unified interface for vision-language multi-task learning. arXiv preprint arXiv:...

    arXiv

  4. [12]

    A simple framework for con- trastive learning of visual representations

    Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In Interna- tional conference on machine learning , pages 1597–

  5. [13]

    Improved baselines with momentum contrastive learning

    Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04297, 2020

    2003 arXiv

  6. [14]

    Qwen2-audio tech- nical report

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al. Qwen2-audio tech- nical report. arXiv preprint arXiv:2407.10759, 2024

    2024 arXiv

  7. [15]

    Simple and controllable music generation

    Jade Copet, Felix Kreuk, Itai Gat, Tal Remez, David Kant, Gabriel Synnaeve, Yossi Adi, and Alexandre D´efossez. Simple and controllable music generation. Advances in Neural Information Processing Systems , 36, 2024

    2024

  8. [16]

    Lauragpt: Listen, attend, under- stand, and regenerate audio with gpt

    Zhihao Du, Jiaming Wang, Qian Chen, Yunfei Chu, Zhifu Gao, Zerui Li, Kai Hu, Xiaohuan Zhou, Jin Xu, Ziyang Ma, et al. Lauragpt: Listen, attend, under- stand, and regenerate audio with gpt. arXiv preprint arXiv:2310.04673, 2023

    arXiv

  9. [17]

    The llama 3 herd of models

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Dwyer, J

    B. Dwyer, J. Nelson, T. Hansen, et al. Roboflow (ver- sion 1.0). Available from https://roboflow. com, 2024. Software

    2024

  11. [19]

    Llama-omni: Seam- less speech interaction with large language models

    Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama-omni: Seam- less speech interaction with large language models. arXiv preprint arXiv:2409.06666, 2024

    2024 arXiv

  12. [20]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 6904–6913, 2017

    2017

  13. [21]

    Aligning ai with shared human values

    Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning ai with shared human values. Proceedings of the International Conference on Learning Repre- sentations (ICLR), 2021

    2021

  14. [22]

    Measuring massive multitask language under- standing

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language under- standing. Proceedings of the International Conference on Learning Representations (ICLR), 2021

    2021

  15. [23]

    Gaussian error linear units (gelus)

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016

    2016 arXiv

  16. [24]

    Scaling up visual and vision- language representation learning with noisy text su- pervision

    Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- language representation learning with noisy text su- pervision. In International conference on machine learning, pages 4904–49...

    2021

  17. [25]

    Chat-univi: Unified visual representation empowers large language models with image and video understanding

    Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xi- aochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 137...

    2024

  18. [26]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin John- son, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. ...

    2017

  19. [27]

    Lisa: Reasoning segmentation via large language model

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579–9589, 2024. 11

    2024

  20. [28]

    Seed-bench: Benchmark- ing multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmark- ing multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  21. [29]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large lan- guage models. In International conference on ma- chine learning, pages 19730–19742. PMLR, 2023

    2023

  22. [30]

    Competition-level code generation with alpha- code

    Yujia Li, David Choi, Junyoung Chung, Nate Kush- man, Julian Schrittwieser, R ´emi Leblond, Tom Ec- cles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. Competition-level code generation with alpha- code. Science, 378(6624):1092–1097, 2022

    2022

  23. [31]

    Microsoft coco: Com- mon objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Com- mon objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzer- land, September 6-12, 2014, Proce...

    2014

  24. [32]

    Improved baselines with visual instruction tun- ing

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tun- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 26296–26306, 2024

    2024

  25. [33]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Ad- vances in Neural Information Processing Systems , volume 36, pages 34892–34916. Curran Associates, Inc., 2023

    2023

  26. [34]

    Mmbench: Is your multi-modal model an all-around player? In Eu- ropean Conference on Computer Vision , pages 216–

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In Eu- ropean Conference on Computer Vision , pages 216–

  27. [35]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR 2019), 2019

    2019

  28. [36]

    Learn to explain: Mul- timodal reasoning via thought chains for science ques- tion answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Pe- ter Clark, and Ashwin Kalyan. Learn to explain: Mul- timodal reasoning via thought chains for science ques- tion answering. Advances in Neural Information Pro- cessing Systems, 35...

    2022

  29. [37]

    Cheap and quick: Effi- cient vision-language instruction tuning for large lan- guage models

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xi- aoshuai Sun, and Rongrong Ji. Cheap and quick: Effi- cient vision-language instruction tuning for large lan- guage models. Advances in Neural Information Pro- cessing Systems, 36, 2024

    2024

  30. [38]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on com- puter vision and pattern recognition , pages 3195– 3204, 2019

    2019

  31. [39]

    Gaia: a benchmark for general ai assistants

    Gr ´egoire Mialon, Cl ´ementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983, 2023

    2023 arXiv

  32. [40]

    Foundation models for generalist medical artificial intelligence

    Michael Moor, Oishi Banerjee, Zahra Shakeri Hos- sein Abad, Harlan M Krumholz, Jure Leskovec, Eric J Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelligence. Nature, 616(7956):259–265, 2023

    2023

  33. [41]

    OpenAI. Gpt-4. Available at https://openai. com/gpt-4 , 2024. Model used for dataset genera- tion

    2024

  34. [42]

    Hello, gpt-4o, 2024

    OpenAI. Hello, gpt-4o, 2024. Accessed: 2024-10-21

    2024

  35. [43]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, ...

    2021

  36. [44]

    Ro- bust speech recognition via large-scale weak supervi- sion

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. Ro- bust speech recognition via large-scale weak supervi- sion. In International conference on machine learn- ing, pages 28492–28518. PMLR, 2023

    2023

  37. [45]

    Language-based action concept spaces improve video self-supervised learning

    Kanchana Ranasinghe and Michael S Ryoo. Language-based action concept spaces improve video self-supervised learning. Advances in Neural Information Processing Systems , 36:74980–74994, 2023

    2023

  38. [46]

    Learning to localize objects improves spatial reason- ing in visual-llms

    Kanchana Ranasinghe, Satya Narayan Shukla, Omid Poursaeed, Michael S Ryoo, and Tsung-Yu Lin. Learning to localize objects improves spatial reason- ing in visual-llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pages 12977–12987, 2024

    2024

  39. [47]

    Audiopalm: A large language model that can speak and listen

    Paul K Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zal´an Borsos, F´elix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, et al. Audiopalm: A large language model that can speak and listen. arXiv preprint arXiv:2306.12925, 2023

    2023 arXiv

  40. [48]

    Laion-5b: An open large- scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large- scale dataset for training next generation image-text models. Advances in Neural Inform...

    2022

  41. [49]

    Hugginggpt: Solv- ing ai tasks with chatgpt and its friends in hugging face

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solv- ing ai tasks with chatgpt and its friends in hugging face. Advances in Neural Information Processing Sys- tems, 36, 2024

    2024

  42. [50]

    Salmonn: Towards generic hearing abilities for large language models

    Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289, 2023

    2023 arXiv

  43. [51]

    Movieqa: Understanding stories in movies through question-answering

    Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. Movieqa: Understanding stories in movies through question-answering. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 4631–4640, 2016

    2016

  44. [52]

    Llama: Open and effi- cient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth ´ee Lacroix, Baptiste Rozi `ere, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  45. [53]

    Llama 2: Open foundation and fine- tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09288, 2023

    2023 arXiv

  46. [54]

    Efficient utilization of large pre-trained models for low resource asr

    Peter Vieting, Christoph L ¨uscher, Julian Dierkes, Ralf Schl¨uter, and Hermann Ney. Efficient utilization of large pre-trained models for low resource asr. In 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), pages 1–5. IEEE, 2023

    2023

  47. [55]

    Neural codec language models are zero-shot text to speech synthe- sizers

    Chengyi Wang, Sanyuan Chen, Yu Wu, Ziqiang Zhang, Long Zhou, Shujie Liu, Zhuo Chen, Yanqing Liu, Huaming Wang, Jinyu Li, et al. Neural codec language models are zero-shot text to speech synthe- sizers. arXiv preprint arXiv:2301.02111, 2023

    2023 arXiv

  48. [56]

    Blsp: Bootstrapping language- speech pre-training via behavior alignment of continu- ation writing

    Chen Wang, Minpeng Liao, Zhongqiang Huang, Jin- liang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. Blsp: Bootstrapping language- speech pre-training via behavior alignment of continu- ation writing. arXiv preprint arXiv:2309.00916, 2023

    2023 arXiv

  49. [57]

    Visionllm: Large lan- guage model is also an open-ended decoder for vision- centric tasks

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large lan- guage model is also an open-ended decoder for vision- centric tasks. Advances in Neural Information Pro- cessing Systems, 36, 2024

    2024

  50. [58]

    Finetuned lan- guage models are zero-shot learners

    Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M Dai, and Quoc V Le. Finetuned lan- guage models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021

    2021 arXiv

  51. [59]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural informa- tion processing systems, 35:24824–24837, 2022

    2022

  52. [60]

    Hard prompts made easy: Gradient-based discrete opti- mization for prompt tuning and discovery

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete opti- mization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36, 2024

    2024

  53. [61]

    Mini-omni: Language models can hear, talk while thinking in streaming

    Zhifei Xie and Changqiao Wu. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725, 2024

    2024 arXiv

  54. [62]

    Mm-bigbench: Evaluat- ing multimodal models on multimodal content com- prehension tasks

    Xiaocui Yang, Wenfang Wu, Shi Feng, Ming Wang, Daling Wang, Yang Li, Qi Sun, Yifei Zhang, Xiaom- ing Fu, and Soujanya Poria. Mm-bigbench: Evaluat- ing multimodal models on multimodal content com- prehension tasks. arXiv preprint arXiv:2310.09036 , 2023

    arXiv

  55. [63]

    Tree of thoughts: Deliberate problem solving with large language models

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Informa- tion Processing Systems, 36, 2024

    2024

  56. [64]

    From image descriptions to visual de- notations: New similarity metrics for semantic in- ference over event descriptions

    Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual de- notations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics , 2:67–78, 2014

    2014

  57. [65]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  58. [66]

    Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A mas- sive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on...

    2024

  59. [67]

    Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities

    Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000, 2023. 13

    2023 arXiv

  60. [68]

    Llama-adapter: Efficient fine- tuning of language models with zero-init attention

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hong- sheng Li, and Yu Qiao. Llama-adapter: Efficient fine- tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023

    2023 arXiv

  61. [69]

    Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language mod- els

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-Yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language mod- els. Advances in Neural Information Processing Sys- tems, 36:5168–5191, 2023

    2023

  62. [70]

    P Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as- a-judge with mt-bench and chatbot arena, 2023

    2023

  63. [71]

    Minigpt-4: Enhancing vision- language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 14

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.