Pith. sign in

REVIEW 3 major objections 7 minor 22 references

Closing the loop between evaluation and data generation lets a multimodal model fix its own weak spots: three rounds of targeted GPT-4 data lift LLaVA-1.5 by 2.53 and MiniGPT4-v2 by 14.62 SEED points.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

A benchmark-guided loop that generates targeted training questions from a model's own mistakes improves vision-language model scores over three rounds.

T0 review reviewed 2026-08-02 challenge →

load-bearing objection A useful closed-loop data-generation system with a real circularity problem: the same benchmark guides the loop and headlines the gains, so the general-capability claim is only partially supported. the 3 major comments →

arxiv 2607.15299 v1 pith:VSIYJ2YX submitted 2026-07-07 cs.MM cs.CVcs.LG

MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation

Zhiyuan Zhao , Bin Wang , Linke Ouyang , Yiqi Lin , Pan Zhang , Xiaoyi Dong , Jiaqi Wang , Conghui He This is my paper
classification cs.MM cs.CVcs.LG
keywords multimodal large language modelsinstruction tuningdata generationclosed-loop learningbad-case samplingbenchmark-driven refinementGPT-4 data synthesisSEED-Bench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that instruction-tuning data for multimodal LLMs should not be collected in isolation from benchmarking. It proposes a closed loop: evaluate the model, collect its wrong answers per capability type, sample the weakest types adaptively, have GPT-4 generate targeted questions in that style, fine-tune, and repeat. Results across three rounds show steady gains—LLaVA-1.5 SEED-Bench rises from 66.04 to 68.57, MiniGPT4-v2 from 49.21 to 63.83—using less data than prior static collections (220k vs. 1.5M samples). The load-bearing premise is that the loop's guidance signal (SEED-Bench bad cases and question formats) transfers to genuine capability rather than benchmark familiarity; the paper's own ablation shows the multiple-choice format drives most of the SEED/MMBench gains while direct-answer data is better on GQA.

Core claim

In its own terms, the paper discovers that a closed loop—evaluation, weakness sampling, targeted generation, retraining—converts a benchmark's error signal into useful training data. The key operation is Adaptive Bad-case Sampling (ABS): for each of nine SEED-Bench capability types, the sampling probability is r_i = sqrt(1 - accuracy_i), so weaker types get more generated questions; representative wrong answers of that type become in-context examples for GPT-4, which also receives dense region descriptions and OCR text from Visual Genome images. Three iterations of this loop improved both tested models across most benchmarks, and data generated for one model transferred to another, suggestin

What carries the argument

The engine is the loop and its steering mechanism: a bad-case pool per fine-grained capability, duplicate reduction via SpaCy similarity, and ABS's adaptive sampling ratio r_i = sqrt(1 - a_i) that allocates more generated data to the dimensions where the model scores lowest. GPT-4 is prompted with the question type definition (borrowed from SEED-Bench), in-context examples sampled from the pool, and rich image information (Visual Genome region descriptions plus PaddleOCR results for text tasks); it emits both a direct answer and a shuffled four-choice multiple-choice format. This converts evaluation errors into new training data aimed at the specific weakness.

Load-bearing premise

The loop's guidance signal and its headline evaluation are the same benchmark (SEED-Bench): wrong answers, question types, and definitions all come from SEED-Bench, so the measured gains may reflect growing familiarity with SEED's multiple-choice format rather than improved general multimodal capability.

What would settle it

Run the identical closed loop but with a different evaluation benchmark serving as the source of bad cases and question definitions (say, MMBench), then test on SEED-Bench. If SEED-Bench gains shrink to near zero while MMBench gains remain large, the improvement is benchmark-specific format learning; if SEED-Bench also improves, the loop produces transferable capability. A cheaper probe: ablate the MC format inside the loop and check whether SEED gains persist when both training and test are open-ended QA.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the loop works, model improvement becomes automatic and targeted: no human annotation is needed between rounds, and each round's evaluation re-runs on the updated model.
  • Data volume matters less than targeting: 220k incremental samples outperformed 1.5M generic SVIT data on several benchmarks.
  • Generated data generalizes across models (e.g., LLaVA-1.5's data improves MiniGPT4-v2), so loops can be reused for different backbones.
  • The multiple-choice format is largely responsible for gains on MC benchmarks like SEED and MMBench, while direct-answer data improves GQA—so format choice is a design lever.
  • Gains saturate after a few rounds, with the paper attributing this to model-side limits like input resolution, implying the loop shifts the bottleneck from data to architecture.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The loop as described risks chasing benchmark format: because bad cases, question types, and definitions all come from SEED-Bench, the improved SEED scores may partly reflect overfitting to that benchmark's question style rather than general multimodal ability; a cleaner test would retrain on loop data but evaluate on a holdout benchmark built from different question templates.
  • ABS's r_i = sqrt(1 - a_i) is one of many possible allocation rules; one could test alternatives (e.g., proportional to expected gain, or uncertainty-weighted) to see if the specific square-root choice matters.
  • A natural extension is to run the loop with a benchmark whose answer format is open-ended rather than multiple choice, which would decouple format effects from capability effects and test the paper's claimed 'targeted capability' improvement.
  • The loop's reliance on GPT-4 as generator means data quality is capped by GPT-4's ability to interpret region captions and OCR; an OCR error or ambiguous region description would propagate into training data—the paper's confidence-based filtering ('only when confident') is an implicit safeguard that could be quantified.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper proposes MLLM-DataEngine, a closed-loop system that iteratively improves a multimodal large language model (MLLM) by: (1) evaluating the model on SEED-Bench and collecting incorrectly answered questions ('bad cases') per fine-grained capability dimension; (2) using the proposed Adaptive Bad-case Sampling (ABS) scheme, with sampling ratio r_i = sqrt(1-a_i), to select question types and in-context examples; (3) prompting GPT-4 to generate new instruction-tuning data (both direct-answer and multiple-choice formats) from Visual Genome region descriptions and OCR output; and (4) fine-tuning the model on the original plus newly generated data, then repeating the cycle. Experiments on LLaVA-1.5 and MiniGPT4-v2 show improvements on SEED-Bench (from 66.04 to 68.57 and from 49.21 to 63.83, respectively) as well as smaller gains on MMBench, MME, and some VQA benchmarks, using less synthetic data than LRV/SVIT. Ablations isolate the effect of ABS versus uniform sampling and of the multiple-choice format, and a cross-model transfer study (A→B) suggests the generated data generalize to other architectures.

Significance. The closed-loop idea, using evaluation feedback to drive targeted data generation, is timely and potentially impactful for MLLM instruction tuning. The paper provides a concrete implementation, releases code/data, and reports careful ablations, including a direct comparison of DA and MC formats that honestly exposes the format tradeoff. If the observed gains reflect genuine capability improvements, the approach would be a strong contribution to data-centric AI. However, the central claim that the loop improves model capability 'in a targeted and automatic manner' is weakened by the fact that the feedback signal (bad cases, question-type definitions, in-context examples) and the headline evaluation are both SEED-Bench, making benchmark overfitting a plausible alternative explanation. The absence of multiple runs and the ad hoc sampling transform further reduce confidence. These issues are fixable with additional experiments and analysis, and the paper's core idea remains defensible.

major comments (3)
  1. [§II-A, §II-B, Supp. §I; Table I] The loop is guided by SEED-Bench at every step: bad cases are collected from SEED-Bench (§II-A), the sampling ratio r_i = sqrt(1-a_i) is computed from per-dimension SEED-Bench accuracies (§II-B), question-type definitions are 'acquired from SEED Benchmark' (Supp. §I), and GPT-4 is prompted to emit SEED-style multiple-choice questions (§II-C). The headline result (Table I) is an improvement on SEED-Bench itself. This circularity means the gains could reflect overfitting to SEED-Bench's format and question distribution rather than general capability. Table III is consistent with this concern: DA-only outperforms DA+MC on GQA (58.60 vs 58.02 at Round 3), while MC drives the SEED/MMBench gains; non-MC benchmarks show only marginal gains (e.g., LLaVA-1.5 GQA +0.75, VizWiz +3.83, VQAv2 +0.62). The authors should provide a held-out evaluation—e.g., use a different benchmark for guidance and eva
  2. [Tables I–II, §III-B] All results are reported as single numbers with no variance estimates. The improvements between rounds (e.g., LLaVA-1.5 SEED 66.04→67.72→68.30→68.57) are small relative to plausible run-to-run variation in SFT/LoRA fine-tuning, especially since Algorithm 1 includes RandomSelect and ABS samples randomly. The cross-model MiniGPT4-v2 gains are large (+14.62) but come with data volumes varying per round (100k/180k/270k), leaving open the question of what fraction is due to data scale vs. the closed-loop guidance. The authors should run multiple seeds (or at least report standard deviations) for the baseline and at least one full loop, and ideally for the main comparison against LRV/SVIT.
  3. [§II-B, Algorithm 1] The sampling transform r_i = sqrt(1-a_i) is introduced without derivation or ablation. It is the core of ABS, so the choice of functional form matters: uniform sampling already yields large gains in Fig. 4, and the comparison shown uses fixed 90K sampled data. The paper does not demonstrate that this particular transform is better than, say, r_i proportional to 1-a_i, a softmax over (1-a_i), or pure uniform sampling with an accuracy-based filter. A sensitivity analysis over the transform and over the duplicate-similarity threshold (0.9 in Algorithm 1) is needed to rule out that the ABS gains are an artifact of tuning these free parameters.
minor comments (7)
  1. [§III-B, Tables I-II] Clarify whether the 'Data Amount' column is cumulative or per-round incremental. The text says 'incremental data generated in the current and the previous rounds,' but the row labels 'Round1 80k, Round2 170k, Round3 220k' suggest cumulative; if so, the per-round incremental amounts are 80k/90k/50k for LLaVA-1.5, which is a different story than 80k/170k/220k.
  2. [Algorithm 1] The similarity function 'spacy' is undefined; specify the exact similarity measure (e.g., token-based Jaccard) and the rationale for the 0.9 threshold.
  3. [Supp. Fig. 3] The prompt template repeats 'Always answer as if you are directly looking at the image' twice; remove the duplicate.
  4. [Supp. §III, Supp. Table I] Typos: 'anylysis' should be 'analysis', 'eaxmple' should be 'example', 'Understading' should be 'Understanding'.
  5. [Introduction, Ref. [15]] Reference [15] is 'Segment Anything,' which is not a benchmark and does not support the claim that current benchmarks 'can point out the model weakness'; likely a citation error.
  6. [Tables IV-VII] Indicate in captions that the evaluation benchmarks are SEED-Bench and MMBench, consistent with the main tables, to make clear that the transfer study does not break the SEED-Bench circularity.
  7. [Supp. §II-B] For MiniGPT4-v2, the number of training epochs differs per round (10/20/30); clarify how this is matched to the data amount growth and whether the baseline was trained with the same number of epochs.

Circularity Check

1 steps flagged

Same-benchmark feedback loop: SEED-Bench supplies the bad cases and sampling ratios that guide data generation, and SEED-Bench is also the headline evaluation metric, so part of the reported gain is benchmark/format alignment rather than independent capability.

specific steps
  1. fitted input called prediction [Sec. II-A, Sec. II-B; headline results in Sec. III-B (Tables I-II); see also Supp. Sec. I]
    "we choose SEED-Bench [13] as the evaluation benchmark ... The sampling ratio for each question type is based on the corresponding evaluation dimensions scores, which is represented by r_i = √(1−a_i), where r_i is the sampling ratio of ith question type and a_i denotes the accuracy attained by MLLM in ith question type."

    The guidance signal a_i is the per-dimension SEED-Bench accuracy, and the paper's headline result is the same SEED-Bench metric: LLaVA-1.5 improves 66.04→68.57 and MiniGPT4-v2 improves 49.21→63.83. Thus the data-selection weights and bad cases are fitted to the exact benchmark later reported as the main success. The loop is further closed by generating SEED-style multiple-choice questions (Supp. I: 'Question Types and its Definition are acquire from SEED Benchmark'). Table III shows that the MC format is the main driver of SEED/MMBench gains while DA-only is better on GQA, so a substantial part of the headline SEED gain reflects format/benchmark alignment with the selection signal, not a general capability increase. This is partial circularity: gains on MMBench, MME, VQA and the cross-mode

full rationale

The strongest circularity is not a self-citation chain or a formal identity but a closed feedback loop: SEED-Bench defines the weakness signal (bad cases and per-dimension accuracies), guides the sampling ratios, supplies the question-type definitions, and is then used as the primary evaluation benchmark. The reported SEED-Bench improvement is therefore partly a measure of how well the generated data matches the SEED-Bench format and question taxonomy, which is exactly what the selection procedure optimizes. However, the paper is not fully circular: it also reports gains on MMBench, MME, and several VQA benchmarks, and the cross-model transfer experiments show the generated data helps a different model. Table III is also transparent about the format effect, showing DA-only data is better on GQA. I found no load-bearing self-citation: references to the authors' prior work are contextual and the benchmark is external. The central claim still has independent content beyond the fitted feedback loop, so the score is moderate rather than high.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 0 invented entities

The ledger shows the result rests on two classes of assumptions: SEED-Bench as a valid proxy for capability, and GPT-4/Visual-Genome as a correct source of new training labels. No new physical entities are introduced; the free parameters are design choices in the loop.

free parameters (4)
  • Sampling ratio transform r_i = sqrt(1 - a_i) = r_i = sqrt(1 - a_i), where a_i is SEED-Bench accuracy for question type i
    Hand-chosen mapping from accuracy to sampling weight. No ablation or sensitivity analysis; alternative transforms could change data mix and results. Introduced in Sec. II-B.
  • Number of in-context examples per prompt = 10
    Ten bad-case examples are selected per question type (Sec. II-B). No study of the effect of this count on generation quality.
  • Duplicate-removal similarity threshold = 0.9 (spaCy similarity)
    Algorithm 1 removes bad cases with similarity > 0.9. Threshold is arbitrary; affects pool diversity and subsequent generation.
  • Per-round incremental data volume = 80k/170k/220k (LLaVA-1.5); 100k/180k/270k (MiniGPT4-v2)
    Data amounts per round are chosen without stated criterion; cumulative data growth co-varies with number of rounds, making round effects and data-scale effects hard to separate (Tables I-II).
axioms (4)
  • domain assumption SEED-Bench scores reliably measure the model's true multimodal capabilities across nine dimensions.
    Used to define weaknesses and success; if SEED is not a valid proxy, the targeted data may not address real deficiencies (Sec. II-A).
  • domain assumption GPT-4 can generate accurate instruction-tuning data from Visual Genome region descriptions and PaddleOCR outputs, and the filtering step removes all harmful errors.
    No human verification of generated answers; a single wrong label in training data could teach incorrect associations (Sec. II-C).
  • domain assumption Visual Genome region descriptions and OCR results supply enough image information to generate correct questions and answers.
    Used as the sole image content in prompts; lossy text representation may cause hallucinated or inaccurate questions (Sec. II-B, prompt template in Fig. 3).
  • domain assumption Bad cases from SEED-Bench are representative of the model's weaknesses rather than artifacts of benchmark wording or choices.
    Bad case pool is built from SEED-Bench errors and used as in-context examples; benchmark-specific biases propagate to generated data (Sec. II-A/II-B).

reviewed 2026-08-02 · how reviews work

0 comments
Cite this review

Pith. "Pith review of MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation." pith.science (2026). https://pith.science/paper/VSIYJ2YX

@misc{pith2026260715299,
  author       = {Pith},
  title        = {Pith review of: MLLM-DataEngine: Closing the Loop of Multimodal Instruction Tuning Data Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSIYJ2YX}},
  note         = {Machine review of arXiv:2607.15299}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

In this paper, we propose MLLM-DataEngine, a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop iteration, the MLLM-DataEngine first analyzes the weakness of the model based on the evaluation results, then generates a proper incremental dataset for the next training iteration, and enhances the model capability iteratively. Compared with previous instruction fine-tuning dataset collection methods which are separate from the benchmarking, MLLM-DataEngine shows better targeting and can improve MLLMs's capabilities more effectively. Firstly, we propose an Adaptive Bad-case Sampling module, which can effectively analyze model weakness based on the benchmarking results and adjust the generation of incremental datasets flexibly. Secondly, in order to ensure high-quality data for specific capability types, the most representative in-context examples and abundant information are provided to GPT-4, which helps GPT-4 fully comprehend the model's weakness and further guarantees high-quality generated data. Through extensive experiments, we find MLLM-DataEngine could boost the MLLMs capability in a targeted and automatic manner without human participants. We hope MLLM-DataEngine could be a general solution for the following MLLMs data curation. Code, data, and model are available at https://github.com/opendatalab/MLLM-DataEngine.

Figures

Figures reproduced from arXiv: 2607.15299 by Bin Wang, Conghui He, Jiaqi Wang, Linke Ouyang, Pan Zhang, Xiaoyi Dong, Yiqi Lin, Zhiyuan Zhao.

Figure 1
Figure 1. Figure 1: Comparison of existing methods and our proposed MLLM-DataEngine. Existing instruct tuning data generation methods [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of proposed MLLM-DataEngine. The whole process is divided into 4 steps. (1) [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Example of generated data for four out of nine fine-grained abilities. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Comparison between uniform sampling and Adaptive [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 1
Figure 1. Figure 1: Results of each capability of MiniGPT4-v2 during the [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Words distributions analysis. Generated instruct data aligns closely to the given question type. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 1
Figure 1. Figure 1: V. QUALITY EXAMPLES IN MLLM-DATAENGINE GENERATED DATA High-quality and diverse examples generated by MLLM￾DataEngine are demonstrated [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Prompt Template for data generation in MLLM-DataEngine. [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Examples of MLLM-DataEngine generated data in Scene Understanding, Instance Identity, and Instance Attribute. [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Examples of MLLM-DataEngine generated data in Instance Localization, Instance Counting, and Spatial Relation. [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Examples of MLLM-DataEngine generated data in Instance Interaction, Visual Reasoning, and Text Recognition. [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 4 linked inside Pith

  1. [1]

    Making the V in VQA matter: Elevating the role of image understanding in visual question answering,

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, and et al., “Making the V in VQA matter: Elevating the role of image understanding in visual question answering,” inCVPR, 2017

    2017

  2. [2]

    OK- VQA: A visual question answering benchmark requiring external knowl- edge,

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and et al., “OK- VQA: A visual question answering benchmark requiring external knowl- edge,” inCVPR, 2019

    2019

  3. [3]

    A-OKVQA: A benchmark for visual question answering using world knowledge,

    Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, and et al., “A-OKVQA: A benchmark for visual question answering using world knowledge,” inECCV, 2022

    2022

  4. [4]

    GQA: A new dataset for real-world visual reasoning and compositional question answering,

    Drew A. Hudson and Christopher D. Manning, “GQA: A new dataset for real-world visual reasoning and compositional question answering,” inCVPR, 2019

    2019

  5. [5]

    OCR- VQA: visual question answering by reading text in images,

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and et al., “OCR- VQA: visual question answering by reading text in images,” inICDAR, 2019

    2019

  6. [6]

    Textcaps: A dataset for image captioning with reading comprehension,

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and et al., “Textcaps: A dataset for image captioning with reading comprehension,” inECCV, 2020

    2020

  7. [7]

    Visual instruction tuning,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee, “Visual instruction tuning,” inNeurIPS, 2024

    2024

  8. [8]

    Sharegpt,

    “Sharegpt,” 2023, https://sharegpt.com/

    2023

  9. [9]

    Visual genome: Connecting language and vision using crowdsourced dense image anno- tations,

    Ranjay Krishna, Yuke Zhu, Oliver Groth, and et al., “Visual genome: Connecting language and vision using crowdsourced dense image anno- tations,”IJCV, 2017

    2017

  10. [10]

    Generation and comprehension of unambiguous object descriptions,

    Junhua Mao, Jonathan Huang, Alexander Toshev, and et al., “Generation and comprehension of unambiguous object descriptions,” inCVPR, 2016

    2016

  11. [11]

    Refer- itgame: Referring to objects in photographs of natural scenes,

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and et al., “Refer- itgame: Referring to objects in photographs of natural scenes,” in EMNLP, 2014

    2014

  12. [12]

    Seed-bench: Benchmarking multimodal llms with generative comprehension,

    Bohao Li, Rui Wang, Guangzhi Wang, and et al., “Seed-bench: Benchmarking multimodal llms with generative comprehension,”arXiv preprint arXiv:2307.16125, 2023

    Pith/arXiv arXiv 2023

  13. [13]

    Mmbench: Is your multi-modal model an all-around player?,

    Yuan Liu, Haodong Duan, Yuanhan Zhang, and et al., “Mmbench: Is your multi-modal model an all-around player?,”arXiv preprint arXiv:2307.06281, 2023

    Pith/arXiv arXiv 2023

  14. [14]

    Mme: A comprehensive evaluation benchmark for multimodal large language models,

    Chaoyou Fu, Peixian Chen, Yunhang Shen, and et al., “Mme: A comprehensive evaluation benchmark for multimodal large language models,”arXiv preprint arXiv:2306.13394, 2023

    Pith/arXiv arXiv 2023

  15. [15]

    Vizwiz grand challenge: Answering visual questions from blind people,

    Danna Gurari, Qing Li, Abigale J. Stangl, and et al., “Vizwiz grand challenge: Answering visual questions from blind people,” inCVPR, 2018

    2018

  16. [16]

    Learn to explain: Multimodal reasoning via thought chains for science question answer- ing,

    Pan Lu, Swaroop Mishra, Tanglin Xia, and et al., “Learn to explain: Multimodal reasoning via thought chains for science question answer- ing,” inNeurIPS, 2022

    2022

  17. [17]

    Lora: Low-rank adaptation of large language models,

    Edward J. Hu, Yelong Shen, Phillip Wallis, and et al., “Lora: Low-rank adaptation of large language models,” inICLR, 2022

    2022

  18. [18]

    Microsoft COCO: common objects in context,

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, and et al., “Microsoft COCO: common objects in context,” inECCV, 2014

    2014

  19. [19]

    Modeling context in referring expressions,

    Licheng Yu, Patrick Poirson, Shan Yang, and et al., “Modeling context in referring expressions,” inECCV, 2016

    2016

  20. [20]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,

    Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, and et al., “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,” inICCV, 2015

    2015

  21. [21]

    Visual spatial reasoning,

    Fangyu Liu, Guy Emerson, and Nigel Collier, “Visual spatial reasoning,” Transactions of the Association for Computational Linguistics, 2023

    2023

  22. [22]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models,

    Deyao Zhu, Jun Chen, Xiaoqian Shen, and et al., “Minigpt-4: Enhancing vision-language understanding with advanced large language models,” arXiv preprint arXiv:2304.10592, 2023. You are an AI visual assistant that can analyze a single image. You receive following information describing the image: the object caption which describes the object, along with de...

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 2, 2026.