Pith. sign in

REVIEW 4 major objections 5 minor 90 references

Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Instructify turns image metadata into training data with open LLMs, beating GPT-4 pipelines by 3% on average.

desk verdict Solid open-source pipeline, but the headline gains come from training on the test benchmarks—the clean comparison is a modest +0.58. read the letter →

arxiv 2505.18115 v1 pith:UDYOYHLR submitted 2025-05-23 cs.CV

Jacob Hansen , Wei Lin , Junmo Kang , Muhammad Jehanzeb Mirza , Hongyin Luo , Rogerio Feris , Alan Ritter , James Glass
show 1 more author
Leonid Karlinsky
This is my paper
classification cs.CV
keywords visualinstructiontuningmetadataconversionASCIItreeopen-sourceLLMdataqualityLLaVA-NextGPT-4distillationscalablegeneration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Instructify, an open-source pipeline that converts standard image metadata (captions, bounding boxes, QA pairs) into visual instruction-tuning (VisIT) conversations using open-weight LLMs like Gemma 2 27B and LLaMA 3.1 70B. The key claim is that this pipeline can reproduce or improve the quality of existing VisIT datasets that were originally generated by proprietary systems like GPT-4, achieving an average 3% improvement across 12 benchmarks and up to 12% on individual ones. The central mechanism is a hierarchical ASCII-tree representation of grounded spatial annotations, combined with iterative context reduction and LLM-based quality control. That makes high-quality instruction data cheaper, more transparent, and more scalable, since it removes the dependence on closed-source APIs.

What carries the argument

The load-bearing component is the ASCII tree: a hierarchical, indented text representation of image objects, their attributes, positions, sizes, depths, and containment relations, built from bounding boxes and masks (via SAM 2 and Depth Anything v2). This tree is serialized into natural-language factual statements that an LLM can exploit; it reduces conversion error from 18.5% (concatenation) to 9.8% in the LVIS study. Around it, the pipeline uses metadata grouping by image source, QA-to-fact reformatting, an iterative generation loop with context reduction (stopping when 85% of metadata is used), verification against the full context, and prompt-template sampling.

What would settle it

Train LLaVA-Next on LLaVA-mix plus (a) the original GPT-4 LLaVA-Instruct-150K and (b) Instructify's reproduction, controlling for exact sample count, image set, and removing ChartQA, DocVQA, TextVQA, and ScienceQA from both training and evaluation. If the Instructify-trained model no longer matches or exceeds the GPT-4-trained model on held-out benchmarks, or if the gains in Table 1 revert to losses, the central claim of reproduction-or-improvement is falsified. An even sharper test: build an Instructify variant that uses a deliberately non-hierarchical serialization (e.g., flat concatenation) and see whether its benchmark gains persist; the paper predicts they would drop.

Watch

Extended reading notes

Core claim

Instructify shows that converting image metadata into structured text, especially through a hierarchical ASCII tree that preserves object attributes, spatial relations, and depth, then feeding that text to an open LLM with staged prompt sampling and quality filtering, yields visual instruction-tuning data that is as good or better than data generated by proprietary GPT-4-based pipelines. Applied to the same images and metadata sources, models fine-tuned on Instructify-generated instruction sets outperform those trained on the original instruction sets by about 3% on average across 12 benchmarks, with individual gains up to 12%.

Load-bearing premise

The evaluation assumes that LLaVA-Next scores on 12 benchmarks isolate the quality of the candidate instruction set when LLaVA-mix is held fixed. That requires the compared datasets to match in size and composition, and the benchmarks to be independent of training metadata.

Editorial extensions

If this is right

  • VisIT data generation no longer needs closed APIs; open models like Gemma 2 27B suffice for comparable or better instruction sets.
  • The same metadata sources can be replayed through Instructify to reproduce or improve datasets like LLaVA-Instruct-150K, PF-1M, and LLaVAR without the original proprietary code.
  • Scaling instruction data (adding more metadata sources or doubling instruction volume) yields further benchmark gains, suggesting a practical path to dataset expansion for niche domains.
  • The ASCII-tree representation is a reusable technique for converting bounding-box/spatial annotations into LLM-friendly text, with a measured ~47% relative error reduction.
  • The framework's Python-based, file-controlled distributed design supports high-throughput, horizontally scalable data generation for large-scale LMM training.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The approach implies an evaluative shortcut: instead of judging instruction data by downstream benchmark scores alone, one could directly measure factual consistency against the metadata source, as the paper's LVIS error-rate study does; a community-wide version of that metric might make metadata-to-instruction fidelity a standard report card.
  • The paper's finding that Gemma 2 27B outperforms LLaMA 3.1 70B in generation quality suggests that smaller, carefully tuned open models may be better suited for instruction-data synthesis than just larger parameter counts; a systematic sweep over open model families could identify even better generators.
  • If the pipeline's quality hold under stricter evaluation, it would lower the barrier for building VisIT datasets in specialized domains (medical, remote sensing, document understanding) where proprietary API costs are prohibitive, but where modest quantities of high-quality metadata exist.
  • The single-prompt simplification used for PF-1M, plus the observed gains on document-heavy benchmarks, hints that simpler prompt designs combined with rich structured context might transfer better across datasets than bespoke prompts.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces Instructify, a multi-stage open-source pipeline that converts image metadata (captions, bounding boxes, QA pairs) from over 40 public datasets into visual instruction tuning (VisIT) conversations using open-weight LLMs such as Gemma 2 27B and LLaMA 3.1 70B. The pipeline comprises data loading and organization, an ASCII-tree serialization of bounding-box metadata, iterative instruction generation with quality filtering and auto-retry, and prompt management. The authors evaluate the generated instruction sets by fine-tuning LLaVA-Next on LLaVA-mix plus a reproduced instruction set, and they compare against LLaVA-mix plus the original GPT-4-generated instruction sets for LLaVA-Instruct-150K, PF-1M, LLaVAR, and DocOwl DocReasoning. They report average gains of roughly 3% across 12 benchmarks and up to 12% on individual benchmarks, together with ablations on the choice of LLM, the ASCII-tree representation, and quality filtering.

Significance. If the quantitative claims hold, this is a useful contribution: it provides a reproducible, open alternative to closed-source GPT-4-based VisIT data generation, with concrete engineering details (ASCII tree, iterative verification, prompt sampling) and released code. The paper's framing as a systems and reproducibility contribution is appropriate, and the idea of unifying diverse metadata into structured text is likely to be adopted by other groups. However, the headline numbers are currently undermined by evaluation-design issues, most importantly the overlap between training metadata sources and evaluation benchmarks, and by unmatched training-data sizes in the PF-1M comparison. These issues are load-bearing because the central claim is explicitly quantitative ('improving GPT-4 generated VisIT instructions by ~3% on average and up to 12% on individual benchmarks').

major comments (4)
  1. [§A.1, Table 10, Table 2] The headline 'up to 12%' gain is not a controlled measure of instruction quality. The +14 Meta. Sources + 2×Instruct. Data condition in Table 2 adds ST-VQA, Chart-QA, and DocVQA as metadata sources (§A.1 states these make up 0.5% of the training data), and Table 2 then reports +12.88% on ChartQA in that condition. ChartQA, DocVQA, TextVQA, and ScienceQA also appear as evaluation benchmarks in Table 1 and Table 7, and several appear as training metadata in Table 10. Training on ChartQA or DocVQA metadata and then evaluating on those same benchmarks inflates scores independently of the quality of the generated conversations. Please recompute the headline result on a hold-out benchmark set, or explicitly exclude all benchmarks that overlap with training metadata sources; without this, the abstract's 'up to 12%' claim is unsupported.
  2. [§4.1, Table 1] The PF-1M reproduction is not matched in size or composition, so the reported gains conflate Instructify's quality with data quantity and source mixture. Section 4.1 says the authors applied the pipeline to a random subset of 19 out of 37 PF-1M datasets and used a single unified prompt, while Table 1 reports 602K samples for 'Our PF' versus 1M for the original PF-1M. This directly contradicts the paper's statement that the reproductions generate 'the same number of instructions for the same images as in the original sets.' The positive PF-1M gains in Table 1 (e.g., DocVQA +4.52, TextVQA +4.68) could be driven by the smaller, differently mixed training set rather than by Instructify. A matched-size comparison on identical source subsets, or an explicit scaling curve, is required before the reproduction claim can be accepted.
  3. [Tables 1–3] All main results are single training runs without error bars, multiple seeds, or significance tests. Many reported differences are within the run-to-run noise of visual-instruction tuning (e.g., AI2D +0.36, POPE +0.01, SeedBench -0.98 in Table 1; the Gemma2-vs-LLaMA comparison in Table 3 is -0.19). The abstract's '~3% on average' is also not tied to a specific table column: Table 1 does not report column averages, and Table 2 reports +2.56% only for the scaled condition. Please report means and standard deviations over at least 2–3 seeds, or a paired bootstrap over benchmarks, and state explicitly which configuration supports the '~3% average' claim.
  4. [§4.5, Table 8] The quality-control claims are based on very small samples, and one reported result contradicts the accompanying text. Table 5 reports precision 0.83 computed from 30 images, which gives a wide confidence interval and cannot support a strong claim of filtering effectiveness. In Table 8, removing object counting reduces the average error count by 0.41, yet the text states that 'every component contributes.' The authors should report sample sizes, the evaluation prompt and protocol, inter-annotator agreement (for manual or GPT-4o judgments), and correct the over-claim. This is load-bearing because quality control is presented as a key contribution of the pipeline.
minor comments (5)
  1. [Abstract] The abstract says the code 'facilities future metadata-to-VisIT data conversion'; 'facilities' should be 'facilitates.'
  2. [§4.3] The text says the scaled condition yields 'an average improvement of 5×' across the 12 benchmarks, but Table 2 shows the average rising from 52.36% to 54.92%, an improvement of 2.56 percentage points, not 5×; this appears to be a typo and should be corrected.
  3. [§4.1, Table 10] Section 4.1 states that each reproduction uses 'the same number of instructions for the same images,' but Table 10 reports 13,104 images for the LLaVAR reproduction while Table 1 reports 16K samples; please reconcile these numbers.
  4. [Table 4] The bounding-box conversion comparison does not state how many images were manually evaluated; please include the sample size and the annotation protocol.
  5. [Throughout] The model name is rendered inconsistently as 'LLaV A,' 'LLaVA,' and 'LlaV A-Next'; please standardize to 'LLaVA' and 'LLaVA-Next.'

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation: the Instructify pipeline is an empirical data-conversion system evaluated on external benchmarks; the flagged ChartQA/DocVQA overlap is a validity concern, not a circular step.

full rationale

Instructify does not derive a result from its own assumptions. The central comparison is empirical: LLaVA-Next is fine-tuned on LLaVA-mix plus either the original GPT-4-generated VisIT data or Instructify's open-model reproduction, and then scored on 12 benchmarks. No component of Instructify is fitted to those benchmark scores, no load-bearing claim rests on a self-citation, and no equation reduces to its own input. The component ablations (ASCII tree, quality filtering) are assessed by human or GPT-4o checks and are independent of the main claim. The reviewer-flagged ChartQA/DocVQA overlap is real but is evaluation contamination: Section A.1 discloses that ChartQA and DocVQA were training metadata sources in the +14 metadata-sources and DocOwl conditions, while Tables 2 and 7 report large gains on those same benchmarks. This weakens those specific numbers as evidence of instruction quality, but it is a correctness risk, not circularity of the derivation chain. The COCO-only LLaVA reproduction and the overall 12-benchmark average provide external signal, so no significant circularity is present.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim is empirical and rests on several design thresholds and modeling assumptions rather than on fitted physical constants. None of the thresholds are derived from first principles; they are engineering choices. The main independent evidence is the 12-benchmark fine-tuning comparison, which is weakened by benchmark overlap and lack of error bars.

free parameters (5)
  • reduction threshold r_t = 0.85
    Stopping criterion in Algorithm 2; generation terminates once 85% of the metadata has been used. Chosen by design, not derived.
  • minimum information length l_min = 100 characters
    Stopping criterion in Algorithm 2; stops if the remaining context is under 100 characters.
  • ASCII tree thresholds t_s, t_m, t_c = not specified
    Spatial, mask, and containment thresholds in Algorithm 1 determine object merging and hierarchy; exact values are not given in the paper.
  • auto-retry count = 3
    Prompt management retries up to three times when no valid conversation is parsed (Section 3.4).
  • prompt distribution P = not specified
    Probability distribution over prompt templates for mixing instruction styles; the values are not reported.
assumptions (4)
  • domain assumption LLM-generated instructions from serialized metadata are a faithful proxy for image content, so fine-tuning on them transfers to benchmark tasks.
    The entire evaluation assumes generated conversations carry the information in the metadata; this is tested but not proven. See Sections 3.3 and 4.
  • domain assumption SAM2 and Depth Anything V2 provide sufficiently accurate masks and depth to build correct object hierarchies.
    Algorithm 1 uses SAM2 masks and depth maps to decide containment and grouping; errors here propagate into the ASCII tree.
  • domain assumption Reformatting QA pairs into declarative statements preserves semantic content.
    Section 3.2 states this conversion prevents repetition; if facts are lost, generated instructions would lose information.
  • domain assumption LLaVA-mix is a neutral base; only the instruction subset being compared changes.
    Section 4.1 uses LLaVA-mix as a fixed baseline, assuming benchmark differences come from the candidate instruction set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion." pith.science (2026). https://pith.science/paper/UDYOYHLR

@misc{pith2026250518115,
  author       = {Pith},
  title        = {Pith review of: Instructify: Demystifying Metadata to Visual Instruction Tuning Data Conversion},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UDYOYHLR}},
  note         = {Machine review of arXiv:2505.18115}
}
read the original abstract

Visual Instruction Tuning (VisIT) data, commonly available as human-assistant conversations with images interleaved in the human turns, are currently the most widespread vehicle for aligning strong LLMs to understand visual inputs, converting them to strong LMMs. While many VisIT datasets are available, most are constructed using ad-hoc techniques developed independently by different groups. They are often poorly documented, lack reproducible code, and rely on paid, closed-source model APIs such as GPT-4, Gemini, or Claude to convert image metadata (labels) into VisIT instructions. This leads to high costs and makes it challenging to scale, enhance quality, or generate VisIT data for new datasets. In this work, we address these challenges and propose an open and unified recipe and approach,~\textbf{\method}, for converting available metadata to VisIT instructions using open LLMs. Our multi-stage \method features an efficient framework for metadata grouping, quality control, data and prompt organization, and conversation sampling. We show that our approach can reproduce or enhance the data quality of available VisIT datasets when applied to the same image data and metadata sources, improving GPT-4 generated VisIT instructions by ~3\% on average and up to 12\% on individual benchmarks using open models, such as Gemma 2 27B and LLaMa 3.1 70B. Additionally, our approach enables effective performance scaling - both in quantity and quality - by enhancing the resulting LMM performance across a wide range of benchmarks. We also analyze the impact of various factors, including conversation format, base model selection, and resampling strategies. Our code, which supports the reproduction of equal or higher-quality VisIT datasets and facilities future metadata-to-VisIT data conversion for niche domains, is released at https://github.com/jacob-hansen/Instructify.

Figures

Figures reproduced from arXiv: 2505.18115 by the authors.

Figure 1
Figure 1. Overview of Instructify. We present a unified framework that automatically transforms diverse metadata from publicly available datasets into multimodal instruction-tuning conversations. Our approach merges metadata from multiple sources, categorizing it into captions, bounding boxes, and question-answer (QA) pairs. While LLMs effectively convert captions and QA into natural language, we find that grounded annotation… view at source ↗
Figure 2
Figure 2. Pipeline of our method. We start with loading and organizing multiple open-source datasets containing captions, question an￾swers, and bounding boxes as metadata. Next, we reformat the metadata in a unified natural language interface which is then converted into multi-modal conversations. These LLM-generated conversations undergo an iterative refinement procedure involving multiple automated tests for quality improv… view at source ↗
Figure 3
Figure 3. ASCII tree example. Our approach automatically collects all available metadata from the available sources (top right) for the image (left). As detailed in the text, we propose a way to re-organize this information into a hierarchical textual data structure - the ASCII tree (middle right) that is effectively converted by an LLM into a detailed high-quality context comprised of multiple factual statements organized in… view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: ASCII tree example. Our approach automatically collects metadata from different available for the image. The ASCII Tree (top right) is a hierarchical textual data structure that includes the attributes, positions, sizes and depth information of objects in the image. Th…
Figure 5
Figure 5. Figure 5: LLaVA prompt [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]
Figure 6
Figure 6. Figure 6: LLaVAR prompt. Polite Flamingo You are an AI assistant that maintains the natural, helpful interaction style of large language models. Transform the provided information into a flowing conversation that bridges raw annotations and natural dialogue. Essential Rules: 1. …
Figure 7
Figure 7. Figure 7: Polite Flamingo prompt [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: DocOwl prompt [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

90 extracted references · 37 canonical work pages

  1. [1]

    anthropic.com/news/claude-3-family, 2024

    Introducing the next generation of claude.https://www. anthropic.com/news/claude-3-family, 2024. 2

    2024

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024

    Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone.arXiv preprint arXiv:2404.14219, 2024. 7

    arXiv 2024

  3. [3]

    Easyocr: Ready-to-use ocr with 80+ supported languages, 2020

    Jaided AI. Easyocr: Ready-to-use ocr with 80+ supported languages, 2020. Accessed: 2024-11-20. 9, 12

    2020

  4. [4]

    Flamingo: a Visual Language Model for Few-Shot Learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Se- bastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sa- hand Sharifzadeh, Mikolaj Bink...

    arXiv 2022

  5. [5]

    Visual instruction tuning with polite flamingo, 2023

    Delong Chen, Jianfeng Liu, Wenliang Dai, and Baoyuan Wang. Visual instruction tuning with polite flamingo, 2023. 6, 9, 12

    2023

  6. [6]

    X-llm: Bootstrapping ad- vanced large language models by treating multi-modalities as foreign languages.arXiv preprint arXiv:2305.04160, 2023

    Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. X-llm: Bootstrapping ad- vanced large language models by treating multi-modalities as foreign languages.arXiv preprint arXiv:2305.04160, 2023. 3

    arXiv 2023

  7. [7]

    Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Juny- ing Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jian- quan Li, Xiang Wan, and Benyou Wang. Allava: Harness- ing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024. 2, 3

    arXiv 2024

  8. [8]

    A spatial-temporal attention- based method and a new dataset for remote sensing image change detection.Remote Sensing, 12(10), 2020

    Hao Chen and Zhenwei Shi. A spatial-temporal attention- based method and a new dataset for remote sensing image change detection.Remote Sensing, 12(10), 2020. 12

    2020

Show all 90 references
  1. [9]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 2, 3

    2023 arXiv

  2. [10]

    Sharegpt4video: Improving video understand- ing and generation with better captions.arXiv preprint arXiv:2406.04325, 2024

    Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, et al. Sharegpt4video: Improving video understand- ing and generation with better captions.arXiv preprint arXiv:2406.04325, 2024. 2, 3

    2024 arXiv

  3. [11]

    Lawrence Zit- nick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zit- nick. Microsoft coco captions: Data collection and evalu- ation server, 2015. 9, 12

    2015

  4. [12]

    Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedan- tam, Saurabh Gupta, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server.arXiv preprint arXiv:1504.00325, 2015. 3

    2015 arXiv

  5. [13]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprint arXiv:2312.14238, 2023

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks.arXiv preprin...

    2023 arXiv

  6. [14]

    InstructBLIP: Towards General-purpose Vision- Language Models with Instruction Tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards General-purpose Vision- Language Models with Instruction Tuning. InNeurIPS,

  7. [15]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tri- pathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024. 2

    2024 arXiv

  8. [16]

    Vila 2: Vila augmented vila.arXiv preprint arXiv:2407.17453, 2024

    Yunhao Fang, Ligeng Zhu, Yao Lu, Yan Wang, Pavlo Molchanov, Jang Hyun Cho, Marco Pavone, Song Han, and Hongxu Yin. Vila 2: Vila augmented vila.arXiv preprint arXiv:2407.17453, 2024. 2

    2024 arXiv

  9. [17]

    Llama-adapter v2: Parameter-efficient vi- sual instruction model.arXiv preprint arXiv:2304.15010,

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xi- angyu Yue, et al. Llama-adapter v2: Parameter-efficient vi- sual instruction model.arXiv preprint arXiv:2304.15010,

    arXiv

  10. [18]

    Multimodal-gpt: A vision and lan- guage model for dialogue with humans.arXiv preprint arXiv:2305.04790, 2023

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and lan- guage model for dialogue with humans.arXiv preprint arXiv:2305.04790, 2023. 3

    2023 arXiv

  11. [19]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing, 2017

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Ba- tra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answer- ing, 2017. 12

    2017

  12. [20]

    Lvis: A dataset for large vocabulary instance segmentation, 2019

    Agrim Gupta, Piotr Doll ´ar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation, 2019. 3, 12

    2019

  13. [21]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF con- ference on computer vision and pattern recognition, pages 6700–6709, 2019. 3, 12

    2019

  14. [22]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14, pages 235–

    2016

  15. [23]

    A hierarchical approach for generating descriptive image paragraphs, 2017

    Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. A hierarchical approach for generating descriptive image paragraphs, 2017. 12

    2017

  16. [24]

    Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer vision, 123:32–73, 2017

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalan- tidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations.International journal of computer...

    2017

  17. [25]

    Seed-bench: Benchmarking mul- timodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yix- iao Ge, and Ying Shan. Seed-bench: Benchmarking mul- timodal llms with generative comprehension.arXiv preprint arXiv:2307.16125, 2023. 7

    2023 arXiv

  18. [27]

    LLaV A-OneVision: Easy Visual Task Trans- fer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy Visual Task Trans- fer.arXiv preprint arXiv:2408.03326, 2024. 1

    2024 arXiv

  19. [28]

    Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36, 2024

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language- and-vision assistant for biomedicine in one day.Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  20. [29]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.arXiv preprint arXiv:2301.12597, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models.arXiv preprint arXiv:2301.12597, 2023. 1

    2023 arXiv

  21. [30]

    Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv preprint arXiv:2305.06355, 2023. 3

    2023 arXiv

  22. [31]

    M3it: A large-scale dataset towards multi- modal multilingual instruction tuning.arXiv preprint arXiv:2306.04387, 2023

    Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, et al. M3it: A large-scale dataset towards multi- modal multilingual instruction tuning.arXiv preprint arXiv:2306.04387, 2023. 3

    2023 arXiv

  23. [32]

    Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions, 2018

    Qing Li, Qingyi Tao, Shafiq Joty, Jianfei Cai, and Jiebo Luo. Vqa-e: Explaining, elaborating, and enhancing your answers for visual questions, 2018. 12

    2018

  24. [33]

    Evaluating object hallucina- tion in large vision-language models.arXiv preprint arXiv:2305.10355, 2023

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucina- tion in large vision-language models.arXiv preprint arXiv:2305.10355, 2023. 7

    2023 arXiv

  25. [34]

    Visual spatial reasoning, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning, 2023. 12

    2023

  26. [35]

    Mitigating hallucination in large multi-modal models via robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Ya- coob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. InThe Twelfth International Conference on Learning Representa- tions, 2023. 2, 3

    2023

  27. [36]

    Re- moteclip: A vision language foundation model for remote sensing, 2024

    Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Re- moteclip: A vision language foundation model for remote sensing, 2024. 12

    2024

  28. [38]

    LLaV A-Next (LLaV A 1.6).arXiv:2310.03744, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. LLaV A-Next (LLaV A 1.6).arXiv:2310.03744, 2023. 7

    2023 arXiv

  29. [39]

    Visual Instruction Tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual Instruction Tuning. InNeurIPS, 2023. 1, 2, 3, 6, 7, 9, 12

    2023

  30. [40]

    On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023

    Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models.arXiv preprint arXiv:2305.07895, 2023. 7

    2023 arXiv

  31. [41]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521,

  32. [42]

    Cheap and quick: Efficient vision- language instruction tuning for large language models.Ad- vances in Neural Information Processing Systems, 36, 2024

    Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision- language instruction tuning for large language models.Ad- vances in Neural Information Processing Systems, 36, 2024. 3

    2024

  33. [43]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. InProceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019. 3, 12

    2019

  34. [44]

    ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Ena- mul Hoque. ChartQA: A benchmark for question answer- ing about charts with visual and logical reasoning. InFind- ings of the Association for Computational Linguistics: ACL 2022, pages 2263–2279, Dublin, Ireland, 2022. A...

    2022

  35. [45]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 3, 7

    2021

  36. [46]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 12

    2021

  37. [47]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rub `en Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Ap- plications of Computer Vision, pages 1697–1706, 2022. 7, 12

    2022

  38. [48]

    Introducing llama 3.1: Our most capable models to date, 2024

    MetaAI. Introducing llama 3.1: Our most capable models to date, 2024. 2, 7, 8

    2024

  39. [49]

    Ocr-vqa: Visual question answering by reading text in images

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. InICDAR, 2019. 12

    2019

  40. [50]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023. 2

    2023 arXiv

  41. [51]

    Gpt-4 technical report.ArXiv, 2303:08774,

    R OpenAI et al. Gpt-4 technical report.ArXiv, 2303:08774,

  42. [52]

    Im2text: Describing images using 1 million captioned pho- tographs.Advances in neural information processing sys- tems, 24, 2011

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs.Advances in neural information processing sys- tems, 24, 2011. 3

    2011

  43. [53]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazeb- nik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. InPro- ceedings of the IEEE international conference on computer v...

    2015

  44. [54]

    Connecting vision and lan- guage with localized narratives

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and lan- guage with localized narratives. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23– 28, 2020, Proceedings, Part V 16, pages 647–664...

    2020

  45. [55]

    Connecting vision and lan- guage with localized narratives, 2020

    Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and lan- guage with localized narratives, 2020. 12

    2020

  46. [56]

    Learning Transferable Visual Models from Natural Language Supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning Transferable Visual Models from Natural Language Supervi- sion. 2021. 7

    2021

  47. [57]

    Sam 2: Segment anything in images and videos,

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman R¨adle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junt- ing Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Doll´ar, and Christoph Feic...

  48. [58]

    Laion-5b: An open large-scale dataset for training next generation image-text models, 2022

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Lai...

    2022

  49. [59]

    LAION-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Worts- man, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev....

    2022

  50. [60]

    A-okvqa: A benchmark for visual question answering using world knowl- edge, 2022

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowl- edge, 2022. 12

    2022

  51. [61]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. InPro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers...

    2018

  52. [62]

    Math- llava: Bootstrapping mathematical reasoning for multimodal large language models.arXiv preprint arXiv:2406.17294,

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math- llava: Bootstrapping mathematical reasoning for multimodal large language models.arXiv preprint arXiv:2406.17294,

    arXiv

  53. [63]

    Textcaps: a dataset for image captioning with reading comprehension, 2020

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020. 12

    2020

  54. [64]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019. 7, 12

    2019

  55. [65]

    Expressing visual relationships via language,

    Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mo- hit Bansal. Expressing visual relationships via language,

  56. [66]

    Vi- sualmrc: Machine reading comprehension on document im- ages, 2021

    Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Vi- sualmrc: Machine reading comprehension on document im- ages, 2021. 12

    2021

  57. [67]

    Gemma 2: Improving open lan- guage models at a practical size, 2024.URL https://arxiv

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupati- raju, L´eonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram ´e, et al. Gemma 2: Improving open lan- guage models at a practical size, 2024.URL https://arxiv. org/abs/240...

    2024 arXiv

  58. [68]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023. 2

    2023 arXiv

  59. [69]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms.arXiv preprint arXiv:2406.16860, 2024. 2

    2024 arXiv

  60. [70]

    To see is to believe: Prompting gpt-4v for better visual instruction tuning.arXiv preprint arXiv:2311.07574, 2023

    Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4v for better visual instruction tuning.arXiv preprint arXiv:2311.07574, 2023. 2, 3

    2023 arXiv

  61. [71]

    Caption anything: Interactive image description with diverse multi- modal controls.arXiv preprint arXiv:2305.02677, 2023

    Teng Wang, Jinrui Zhang, Junjie Fei, Hao Zheng, Yunlong Tang, Zhe Li, Mingqi Gao, and Shanshan Zhao. Caption anything: Interactive image description with diverse multi- modal controls.arXiv preprint arXiv:2305.02677, 2023. 1

    2023 arXiv

  62. [72]

    Visionllm: Large language model is also an open- ended decoder for vision-centric tasks.Advances in Neural Information Processing Systems, 36, 2024

    Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open- ended decoder for vision-centric tasks.Advances in Neural Information Processing Systems, 36, 2024. 3

    2024

  63. [73]

    Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4.arXiv preprint arXiv:2308.12067, 2023

    Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4.arXiv preprint arXiv:2308.12067, 2023. 3

    2023 arXiv

  64. [74]

    Q-bench: A benchmark for general-purpose foundation models on low-level vision

    Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpose foundation models on low-level vision. arXiv preprint arXiv:2309.14181, 2023. 7

    2023 arXiv

  65. [75]

    Grok-1.5 vision preview, 2024

    XAI. Grok-1.5 vision preview, 2024. 7

    2024

  66. [76]

    Multiinstruct: Im- proving multi-modal zero-shot learning via instruction tun- ing.arXiv preprint arXiv:2212.10773, 2022

    Zhiyang Xu, Ying Shen, and Lifu Huang. Multiinstruct: Im- proving multi-modal zero-shot learning via instruction tun- ing.arXiv preprint arXiv:2212.10773, 2022. 3

    2022 arXiv

  67. [77]

    Depth any- thing v2, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiao- gang Xu, Jiashi Feng, and Hengshuang Zhao. Depth any- thing v2, 2024. 5

    2024

  68. [78]

    mplug-docowl: Modularized multimodal large language model for document understanding.arXiv preprint arXiv:2307.02499, 2023

    Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Yuhao Dan, Chenlin Zhao, Guohai Xu, Chenliang Li, Jun- feng Tian, et al. mplug-docowl: Modularized multimodal large language model for document understanding.arXiv preprint arXiv:2307.02499, 2023. 10, 12

    2023 arXiv

  69. [79]

    mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality.arXiv preprint arXiv:2304.14178, 2023. 1

    2023 arXiv

  70. [80]

    Berg, and Tamara L

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expres- sions, 2016. 3, 12

    2016

  71. [81]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for ex- pert agi. InProceedings of the IEEE/CVF Conference on ...

    2024

  72. [82]

    Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guo- qiang Wei, Yang Wei, Yuchen Zhang, Tao Kong, and Ruihua Song. What matters in training a gpt4-style language model with multimodal inputs? InProceedings of the 2024 Confer- ence of the North American Chapter of the Associa...

    2024

  73. [83]

    Rsvg: Exploring data and models for visual grounding on remote sensing data

    Yang Zhan, Zhitong Xiong, and Yuan Yuan. Rsvg: Exploring data and models for visual grounding on remote sensing data. IEEE Transactions on Geoscience and Remote Sensing, 61: 1–13, 2023. 12

    2023

  74. [84]

    Lmms- eval: Reality check on the evaluation of large multimodal models, 2024

    Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms- eval: Reality check on the evaluation of large multimodal models, 2024. 9

    2024

  75. [85]

    Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112, 2023

    Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Haodong Duan, Songyang Zhang, Shuangrui Ding, et al. Internlm- xcomposer: A vision-language large model for advanced text-image comprehension and composition.arXiv preprint arXiv:2309.15112, 2023. 2

    2023 arXiv

  76. [86]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Ao- jun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention.arXiv preprint arXiv:2303.16199, 2023. 3

    2023 arXiv

  77. [87]

    Hierarchical and robust convolutional neural network for very high-resolution remote sensing object detection

    Yuanlin Zhang, Yuan Yuan, Yachuang Feng, and Xiaoqiang Lu. Hierarchical and robust convolutional neural network for very high-resolution remote sensing object detection. IEEE Transactions on Geoscience and Remote Sensing, 57 (8):5535–5548, 2019. 12

    2019

  78. [88]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 6, 9, 12

    2023 arXiv

  79. [89]

    Svit: Scaling up visual instruction tuning.arXiv preprint arXiv:2307.04087, 2023

    Bo Zhao, Boya Wu, Muyang He, and Tiejun Huang. Svit: Scaling up visual instruction tuning.arXiv preprint arXiv:2307.04087, 2023. 2, 3

    2023 arXiv

  80. [90]

    Chatbridge: Bridging modalities with large language model as a language catalyst.arXiv preprint arXiv:2305.16103, 2023

    Zijia Zhao, Longteng Guo, Tongtian Yue, Sihan Chen, Shuai Shao, Xinxin Zhu, Zehuan Yuan, and Jing Liu. Chatbridge: Bridging modalities with large language model as a language catalyst.arXiv preprint arXiv:2305.16103, 2023. 3

    2023 arXiv

  81. [91]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mo- hamed Elhoseiny. MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models. arXiv preprint arXiv:2304.10592, 2023. 1, 3

    2023 arXiv

  82. [92]

    Visual7w: Grounded question answering in images, 2016

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images, 2016. 12

    2016

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.