Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning a lightweight multimodal model on a small set of self-question-answer-description-caption conversations is claimed to reduce hallucinations by 31.2% and improve zero-shot visual reasoning.

desk verdict The training recipe is plausible and cheap, but the headline hallucination metric is sign-confused as written and the evaluation loop is same-generator, so the main claim does not survive close reading. read the letter →

arxiv 2501.02964 v2 pith:N2HDJET6 submitted 2025-01-06 cs.CV cs.AI

Wanpeng Hu , Haodi Liu , Lin Chen , Feng Zhou , Changming Xiao , Qi Yang , Changshui Zhang This is my paper · ORCID
classification cs.CVcs.AI
keywords SocraticQuestioningmultimodallargelanguagemodelshallucinationmitigationvisualreasoningchain-of-thoughtinstructiontuningCapQAzero-shotgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes Socratic Questioning (SQ), a training-and-inference routine in which a lightweight multimodal language model asks itself a list of questions about an image, answers those questions one by one, and then consolidates the question-answer pairs first into a detailed description and finally into a short caption. The authors claim that adding just under a thousand such multi-turn exchanges—the CapQA dataset—to a standard visual-instruction mix teaches the model to ground its descriptions in fine-grained visual detail and to reason about everyday human activities it was not trained on. On their CapQA benchmark, SQ raises the question-quality metric from 31.5 to 92.3 and improves the hallucination metric by 31.2% relative to the base model, while on the POPE object-existence benchmark it reports the best F1 across random, popular, and adversarial sampling. If the claim holds, hallucination in small models can be reduced by changing the training format alone, without adding parameters, raising image resolution, or introducing extra decoding machinery.

What carries the argument

The load-bearing object is the Socratic loop—self-ask, self-answer, consolidate and organize, summarize and condense—encoded as a structured multi-turn instruction-tuning conversation. The first turn trains the model to generate a question list, the middle turns train it to answer those questions one at a time, and the last two turns train it to produce a detailed description and then a condensed caption. At inference, the same weights run the loop in 3-turn mode or skip to a caption in 1-turn mode. This loop is what the paper credits with forcing the model to retrieve fine-grained visual information before producing its final output, and therefore with reducing hallucinations and improving zero-shot reasoning.

What would settle it

Ask human annotators to score the same CapQA test outputs for visual groundedness and question usefulness against the original images, then compare the SQ model with the base model; if humans do not find roughly the same 31% hallucination reduction, the central claim fails.

Watch

Extended reading notes

Core claim

SQ is a four-step reasoning loop made into a multi-turn conversation: the model first self-asks a set of 5–8 questions targeted at the activity in the image, then self-answers each question with visually grounded VQA responses, then consolidates and organizes the Q&A into a detailed description, and finally summarizes and condenses that description into a caption retaining the core content. All four roles are played by the same lightweight language model, trained jointly; at inference the model can run the whole loop (3-turn mode) or jump straight to the final caption (1-turn mode). The authors argue this loop counters a central cause of hallucination they identify: multimodal models tend to ignore visual features during autoregressive generation, whereas the self-questioning step forces attention back to visual clues before any final claim is made. They report that this formulation, fine-tuned with LoRA on 666K image-text pairs including 882 CapQA samples, transfers zero-shot to six standard benchmarks and reduces object-existence hallucinations on POPE.

Load-bearing premise

The load-bearing premise is that GPT-4's ratings are a valid and unbiased measure of hallucination and question quality, because the same model family generated the training annotations and scores the outputs, and if the judge merely rewards outputs that imitate its own generation style, the reported improvements would be an artifact rather than visual grounding.

Editorial extensions

If this is right

  • If the claims hold, a 7B-parameter multimodal model can gain zero-shot visual reasoning ability from roughly a thousand extra multi-turn conversations, making the annotation budget for instruction tuning much smaller than current practice.
  • Hallucination mitigation would no longer require larger vision encoders, higher input resolution, or decoding-time intervention; a training-format change would suffice.
  • The large jump in question-quality metric (31.5 to 92.3) implies that models can learn to ask informative, answerable questions from very little data, which could be harnessed for other self-guided tasks.
  • The 3-turn inference mode's extra 2.3% reduction in hallucination indicates that running the full question-answer-caption loop at inference is worthwhile for complex questions, while 1-turn mode remains cheaper for simple ones.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the CapQA training labels and the GPT-4 scoring prompt come from the same model family, the reported 31.2% improvement may partly reflect stylistic imitation rather than visual grounding; a human-scored evaluation would be needed to separate the two, and the paper does not provide one.
  • If SQ works by forcing visual attention, its benefit should be sensitive to the order and content of the self-asked questions; a controlled variant that substitutes generic or random questions would isolate the mechanism, but the paper does not run that ablation.
  • The framework suggests a cheap route to domain adaptation beyond human activities: generating CapQA-style self-questioning conversations for other fine-grained domains, such as medical images, satellite imagery, or industrial scenes, could transfer SQ benefits, a direction the paper leaves implicit.
  • The 1-turn versus 3-turn trade-off points toward an adaptive controller that decides whether to invoke self-questioning based on expected problem complexity, something not explored here.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Socratic Questioning (SQ), a multi-round self-questioning framework for lightweight multimodal LLMs. Given an image and a question, the model generates a list of questions, answers them, writes a detailed description, and produces a summarized caption. The authors automatically create a ~1k-image dataset (CapQA) by prompting GPT-4V, fine-tune LLaVA-1.5-7B on CapQA plus the LLaVA-1.5 mix, and evaluate on CapQA, POPE, MMHal, MME, LLaVA-QA90, LLaVA-Bench, ScienceQA-IMG, TextVQA, and VCR/SNLI-VE. The headline result is a 31.2% improvement in a GPT-4-scored hallucination metric (HalS) on CapQA.

Significance. The core idea — training a small MLLM to ask itself fine-grained visual questions before answering — is simple, inexpensive, and potentially useful, and the detailed hyperparameters plus the planned release of CapQA are practical contributions. If the reported hallucination reduction were robust, this would be a valuable low-cost intervention. The external benchmarks (POPE, MMHal, MME) give weak but directionally consistent support. However, the manuscript's own CapQA metric is internally inconsistent and confounded by generator-evaluator overlap, so the central quantitative claim is not currently credible. The paper contains no machine-checked proofs or released code, and the main empirical claims lack error bars.

major comments (4)
  1. [Section 5.1, Eq. (2), Table 6] The HalS score direction is internally inconsistent. Eq. (2) defines HalS = HalSpred/HalSgt and the text states 'higher scores indicating less hallucination,' but the GPT-4 evaluation prompt in Table 6 says 'a lower score indicates fewer hallucinations and better performance.' Under the prompt's literal meaning, an output with fewer hallucinations than the ground-truth label receives a lower raw score, so HalS < 1, and a higher HalS means more hallucination relative to the label. Read literally, the claimed improvement from 69.3 to 90.9 in Table 2 is a regression, not a 31.2% improvement. If the authors intended the raw scores as quality scores (higher = better), the prompt is mis-specified. Table 12's raw columns (LLaVA-1.5 pred 41.0 vs SQ pred 53.0 on the same 30 samples) illustrate the ambiguity. The metric must be redefined and re-run before the headline claim can be evaluated.
  2. [Section 5.1, Section 4.2, Table 6] The CapQA evaluation is a same-generator loop. GPT-4V produced the CapQA training annotations (Section 4.2) and the reference labels used in Eq. (2); the GPT-4 judge scores model outputs against those GPT-4V labels, and the QQS prompt even supplies the expected GPT-4V questions to the judge. A model fine-tuned to imitate GPT-4V's style is therefore rewarded by the same model family that created the data, independent of visual grounding. No human evaluation or correlation with human judgments is reported. This confound must be addressed (e.g., human ratings on a subset or a metric not based on GPT-4V-generated references) before CapQA can support the 31.2% claim.
  3. [Section 5.3, Tables 3 and 4] The independent external evidence is too thin to carry the hallucination claim on its own. POPE F1 gains over LLaVA-1.5 are 1.44, 0.77, and 0.69 points in the random, popular, and adversarial settings; MMHal Average Score moves from 2.04 to 2.16 and Hallucination Rate from 0.61 to 0.57. These are single-run differences, reported without confidence intervals or significance tests. Once the CapQA metric is corrected, the paper needs either repeated runs with variance estimates or a human evaluation to establish that SQ reduces hallucinations.
  4. [Section 4.5, Tables 3, 4, 9-11] The inference mode used for each reported result is not specified. The paper says 1-turn inference suits straightforward problems and 3-turn inference suits complicated ones, and Table 2 shows a non-negligible difference between the two modes (90.9 vs 93.0 HalS). Without stating which mode produced each benchmark number, the comparisons are not reproducible.
minor comments (6)
  1. [Abstract and Section 5.1] 'Our proposed SQ method leads to a 31.2% improvement in the hallucination score' should be reworded to state the metric direction, the baseline, and the sign.
  2. [Section 3.1, References] 'archtecture' should be 'architecture'; 'LLaV A' spacing is inconsistent throughout; 'NeurlPS' in references [26] and [42] should be 'NeurIPS'; references [31] and [32] are duplicated.
  3. [Table 12] The column header is difficult to parse; a table with separate 'run', 'pred/gt', 'gt', and 'pred' columns would clarify the reported numbers.
  4. [Table 10] The CapQA30 HalS of 130.0 for SQ-caponly exceeds 100; the paper should explain whether normalized scores can exceed 100 and what that implies.
  5. [Section 4.2, Table 1] The prompt asks for 5-8 questions and later allows up to 20, while the final format in Table 8 shows 13 questions; the refinement and deduplication procedure should be described.
  6. [Abstract] The GitHub link should point to a released repository, and the dataset download location should be given.

Circularity Check

2 steps flagged · score 6.0 of 10

CapQA's headline metrics (HalS and QQS) are a same-generator loop: GPT-4V produced the training labels, the model was fine-tuned to imitate them, and GPT-4 scores fidelity to those same labels; the QQS metric reduces to a training-fit score by construction, while external benchmarks provide independent but partial support.

  1. fitted input called prediction [Section 4.4.1 (Questions Generation) and Table 6 (Questions Quality prompt), used by Section 5.1 Eq. (2) QQS]
    "The first turn [X1_q, X1_a] is specifically designed to train the LLM to function as a Question Generator... X1_q denotes a carefully crafted prompt requesting the questions generation, while X1_a denotes the list of questions generated upon X1_q."

    The GPT-4 evaluation prompt for QQS tells the judge: 'we will provide additional information about the image and questions (such as the expected questions, human-generated questions, and hints given by annotators)'. Those 'expected questions' are the GPT-4V-generated question lists from Section 4.2, which are exactly the X1_a targets used to train the question-generation turn. The reported QQS jump (31.5 to 92.3) therefore measures how well the model reproduces its own training labels, not an independent property of question diversity or effectiveness. The 'prediction' of question quality is, by construction, a training-fit score against the same teacher that generated the fit target.

  2. fitted input called prediction [Section 4.2 (GPT-4V annotation), Section 4.4.1 (Detailed Description Generation), and Section 5.1 Eq. (2) HalS]
    "HalS = HalSpred/HalSgt (2)... HalSpred represents the average score of all model predictions reviewed by GPT-4, while HalSgt denotes the average score of all labels reviewed by GPT-4. ... we utilize GPT-4v [1] to automatically generate the annotations including a list of questions, corresponding answers, a detailed description and a summarized caption."

    The CapQA ground-truth labels used as HalSgt are GPT-4V's generated detailed descriptions, and the model's detailed-description turn was trained to imitate exactly those labels. GPT-4 then rates both the model output and the GPT-4V label with the same hallucination rubric. The ratio HalS = pred/gt is thus a same-generator fidelity measure: a model that imitates GPT-4V's descriptive style will tend to receive scores close to, or better than, the teacher's own labels. The claimed 31.2% hallucination 'improvement' is therefore not an external check on visual grounding but a measure of stylistic agreement with the model family that produced both the training signal and the evaluation reference.

full rationale

This paper has no load-bearing self-citations: the authors cite standard MLLM baselines and tools (LLaVA, Vicuna, GPT-4), not their own prior results, and no uniqueness theorem is imported from the authors. The central claim that SQ helps on external benchmarks is independently supported: POPE, MMHal, MME, TextVQA, ScienceQA-IMG, and LLaVA-Bench are established benchmarks not generated by the paper's GPT-4V loop, and the reported gains there are small but real (e.g., POPE F1 84.87 vs 84.18 adversarial; MMHal HalR 0.57 vs 0.61). However, the headline CapQA results in Table 2 are partially circular. The QQS metric is the clearest reduction: the evaluator is given the GPT-4V 'expected questions' as reference while the question-generation turn was trained on those same GPT-4V question lists; the 31.5-to-92.3 jump is essentially a memorization/imitation score. The HalS metric is similarly entangled: both numerator and denominator are GPT-4-family ratings of outputs whose training target was GPT-4V's own description style, so the 31.2% improvement measures teacher-style matching rather than independently verified hallucination. I also note an internal inconsistency that compounds the problem: Eq. (2) defines HalS = pred/gt, but Table 6's hallucination prompt says 'a lower score indicates fewer hallucinations and better performance,' while Section 5.1 asserts higher HalS means less hallucination. Under the literal prompt, higher HalS would mean more hallucination relative to the label, potentially reversing the sign of the reported '+31.2%'. This is a definitional/validity flaw rather than a circularity reduction, so I do not count it as a separate circular step, but it further weakens the headline metric. Overall, the CapQA metrics reduce in part to same-generator training-fit evaluations (score contribution toward 6), while the external benchmarks prevent the paper from being fully circular.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claims rest on assumptions about the reliability of GPT-4V as an annotation generator, GPT-4 as a judge, and the representativeness of small hand-picked datasets. These are domain assumptions rather than mathematical axioms, and none is independently validated in the paper.

free parameters (4)
  • CapQA dataset composition = 982 images (882 train, 100 test) from 20 CAP activity classes, 50 clips per class
    Hand-selected activity classes and key-frame extraction; no ablation over class count, clip count, or frame selection. The small test set (100 images) is used for the headline hallucination claim.
  • GPT-4V annotation prompt constraints = 5-8 initial questions, refine up to 20, detailed description <=1000 words, concise <=400 words
    These numbers determine the structure of CapQA and therefore what the model learns; they were not swept or justified empirically.
  • Training mixture and hyperparameters = 666K total (llava_v1_5_mix665k + CapQA_0.9k), batch size 128, lr 2e-5, 1 epoch, LoRA, DeepSpeed stage 3
    Follows LLaVA-1.5 defaults; no experiments isolating the effect of adding 1k samples versus the SQ format.
  • Inference mode selection = 1-turn for straightforward problems, 3-turn for complex problems
    The choice is made per task after observing results, a post hoc selection that makes benchmark comparisons difficult to interpret.
assumptions (5)
  • domain assumption GPT-4V-generated annotations are factually accurate and visually grounded
    Section 4.2 uses GPT-4V to generate all CapQA labels, which are then used as ground truth for training and evaluation. If GPT-4V hallucinates or misses details, the dataset inherits those errors.
  • domain assumption GPT-4 hallucination and question-quality scores are valid proxies for human judgment
    Section 5.1 and Table 6 define HalS and QQS entirely by GPT-4 ratings. No correlation with human raters is reported.
  • domain assumption The CAP dataset activity labels and key frames are correct
    Section 4.1 selects images using CAP video labels; errors would propagate to CapQA and the evaluations.
  • domain assumption External benchmarks (POPE, MMHal, MME, ScienceQA-IMG, TextVQA) are fair and their evaluation protocols are accepted
    Section 5.3 and Appendix C use these benchmarks as evidence; the paper does not contribute to their validity.
  • domain assumption Sampling 5000 examples from VCR and SNLI-VE is representative
    Table 9 compares SQ with IdealGPT on 5000 sampled items; sampling is said to follow IdealGPT but no details or error bars are given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild." pith.science (2026). https://pith.science/paper/N2HDJET6

@misc{pith2026250102964,
  author       = {Pith},
  title        = {Pith review of: Socratic Questioning: Learn to Self-guide Multimodal Reasoning in the Wild},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N2HDJET6}},
  note         = {Machine review of arXiv:2501.02964}
}
read the original abstract

Complex visual reasoning remains a key challenge today. Typically, the challenge is tackled using methodologies such as Chain of Thought (COT) and visual instruction tuning. However, how to organically combine these two methodologies for greater success remains unexplored. Also, issues like hallucinations and high training cost still need to be addressed. In this work, we devise an innovative multi-round training and reasoning framework suitable for lightweight Multimodal Large Language Models (MLLMs). Our self-questioning approach heuristically guides MLLMs to focus on visual clues relevant to the target problem, reducing hallucinations and enhancing the model's ability to describe fine-grained image details. This ultimately enables the model to perform well in complex visual reasoning and question-answering tasks. We have named this framework Socratic Questioning(SQ). To facilitate future research, we create a multimodal mini-dataset named CapQA, which includes 1k images of fine-grained activities, for visual instruction tuning and evaluation, our proposed SQ method leads to a 31.2% improvement in the hallucination score. Our extensive experiments on various benchmarks demonstrate SQ's remarkable capabilities in heuristic self-questioning, zero-shot visual reasoning and hallucination mitigation. Our model and code will be publicly available.

Figures

Figures reproduced from arXiv: 2501.02964 by the authors.

Figure 1
Figure 1. Comparison of Questions Generation on LLaVA-with-SQ and LLaVA. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. SQ network architecture. Note that the two LLM modules correspond to a single LLM. The [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Illustrations of the training (left) and 3-turn inference (right) processes of SQ. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of Questions Generation on LLaVA-with-SQ and LLaVA. [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]
Figure 5
Figure 5. Figure 5: An example of SQ. SQ starts by asking questions about the given image for heuristic self [PITH_FULL_IMAGE:figures/full_fig_p018_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Introspection of Thought Helps AI Agents

    cs.AI 2025-07 conditional novelty 4.0 of 10

    INoT wraps prompts in XML-defined pseudo-code so an LLM simulates two debating agents internally, reporting better scores and lower tokens than seven baselines.

Reference graph

Works this paper leans on

46 extracted references · 18 canonical work pages · cited by 1 Pith paper

  1. [1]

    Chatgpt can now see, hear, and speak https://openai.com/index/chatgpt-can-now-see-hear-and- speak/. 2023. 2, 5

    work page 2023

  2. [2]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint, (2308.12966), 2023. 2, 4, 9

    arXiv 2023

  3. [3]

    Hallucination of multimodal large language models: A survey

    Zechen Bai, Pichao Wang, Tianjun Xiao, Tong He, Han Zongbo, Zheng Zhang, and Zheng Mike Shou. Hallucination of multimodal large language models: A survey. arXiv preprint, (2404.18930), 2024. 4

    arXiv 2024

  4. [4]

    Fine-grained activities of people worldwide, 2022

    Jeffrey Byrne, Greg Castanon, Zhongheng Li, and Gil Ettinger. Fine-grained activities of people worldwide, 2022. 5

    work page 2022

  5. [5]

    Compositional chain-of-thought prompting for large multimodal models

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Compositional chain-of-thought prompting for large multimodal models. arXiv preprint, (2311.17076), 2024. 4 12

    arXiv 2024

  6. [6]

    Measuring and improving chain-of-thought reasoning in vision-language models

    Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measuring and improving chain-of-thought reasoning in vision-language models. arXiv preprint, (2309.04461),

    arXiv

  7. [7]

    Halc: Object hallucination reduction via adaptive focal-contrast decoding

    Zhaorun Chen, Zhuokai Zhao, Hongyin Luo, Huaxiu Yao, Bo Li, and Jiawei Zhou. Halc: Object hallucination reduction via adaptive focal-contrast decoding. arXiv preprint, (2403.00425),

    arXiv

  8. [8]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, and Lewei Lu. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint, (2312.14238), 2023. 4

    arXiv 2023

Show all 46 references
  1. [9]

    Gonzalez, Ion Stoica, and Eric P

    Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 5, 8

    2023

  2. [10]

    Instructblip: Towards general-purpose vision-language models with instruction tuning

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint, (2305.06500), 2023. 2, 9

    2023 arXiv

  3. [11]

    Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding

    Ailin Deng, Zhirui Chen, and Bryan Hooi. Seeing is believing: Mitigating hallucination in large vision-language models via clip-guided decoding. arXiv preprint, (2402.15300), 2024. 4

    2024 arXiv

  4. [12]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  5. [13]

    Mme: A comprehensive evaluation benchmark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint, (2306.13394),

    arXiv

  6. [14]

    Multi-modal latent space learning for chain-of-thought reasoning in language models

    Liqi He, Zuchao Li, Xiantao Cai, and Ping Wang. Multi-modal latent space learning for chain-of-thought reasoning in language models. AAAI, 2023. 3

    2023

  7. [15]

    Incorporating visual experts to resolve the information loss in multimodal large language models

    Xin He, Longhui Wei, Lingxi Xie, and Qi Tian. Incorporating visual experts to resolve the information loss in multimodal large language models. arXiv preprint, (2401.03105), 2024. 4

    2024 arXiv

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Zhu, Yuanzhi Li, Shean Lu, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint, (2106.09685), 2021. 7

    2021 arXiv

  9. [17]

    Vcoder: Versatile vision encoders for multimodal large language models

    Jitesh Jain, Jianwei Yang, and Humphrey Shi. Vcoder: Versatile vision encoders for multimodal large language models. arXiv preprint, (2312.14233), 2023. 4

    2023 arXiv

  10. [18]

    Enhancing multimodal large language models with vision detection models: An empirical study

    Qirui Jiao, Daoyuan Chen, Yilun Huang, Yaliang Li, and Ying Shen. Enhancing multimodal large language models with vision detection models: An empirical study. arXiv preprint, (2401.17981), 2024. 4

    2024 arXiv

  11. [19]

    Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643, 2023. 10

    2023 arXiv

  12. [20]

    Otter: A multi-modal model with in-context instruction tuning

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023. 9

    2023 arXiv

  13. [21]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint, (2301.12597), 2023. 9

    2023 arXiv

  14. [22]

    Silkie: Preference distillation for large visual language models

    Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Kong Lingpeng. Silkie: Preference distillation for large visual language models. arXiv preprint, (2312.10665), 2023. 4 13

    2023 arXiv

  15. [23]

    Grounded language-image pre-training

    Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. Grounded language-image pre-training. In CVPR, 2022. 10

    2022

  16. [24]

    Evaluating object hallucination in large vision-language models

    Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint, (2305.10355), 2023. 8, 9

    2023 arXiv

  17. [25]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint, (2310.03744), 2023. 2, 4, 5, 7, 8, 9, 16

    2023 arXiv

  18. [26]

    Visual instruction tuning.NeurlPS,

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurlPS,

  19. [27]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Cheng, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. NeurlPS, 2022. 8, 9

    2022

  20. [28]

    Kam-cot: Knowledge augmented multimodal chain-of-thoughts reasoning

    Debjyoti Mondal, Suraj Modi, Subhadarshi Panda, Rituraj Singh, and Godawari Sudhakar Rao. Kam-cot: Knowledge augmented multimodal chain-of-thoughts reasoning. AAAI, 2024. 3

    2024

  21. [29]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv preprint, (2303.08774), 2024. 3, 8

    2024 arXiv

  22. [30]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. CVPR, 2019. 8, 9, 17

    2019

  23. [32]

    Align large multimodal models with factually augmented rlhf

    Zhiqing Sun, Sheng Shen, Shengcao Cao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, and Trevor Darrell. Align large multimodal models with factually augmented rlhf. arXiv preprint, (2309.14525), 2023. 9

    2023 arXiv

  24. [33]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. arXiv preprint, (2401.06209),

    arXiv

  25. [34]

    T-sciq: Teaching multimodal chain-of-thought reasoning via mixed large language model signals for science question answering

    Lei Wang, Yi Hu, Jiabang He, Xing Xu, Ning Liu, Hui Liu, and Heng Tao Shen. T-sciq: Teaching multimodal chain-of-thought reasoning via mixed large language model signals for science question answering. AAAI, 2024. 4

    2024

  26. [35]

    mplug-owl: Modularization empowers large language models with multimodality, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chaoya Jiang, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models w...

    2023

  27. [36]

    Woodpecker: Hallucination correction for multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang, Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large language models. arXiv preprint arXiv:2310.16045, 2023. 8, 9

    arXiv

  28. [37]

    Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback

    Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He, Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun. Rlhf-v: Towards trustworthy mllms via behavior alignment from fine-grained correctional human feedback. arXiv preprint, (2312.00849), 2023. 4

    2023 arXiv

  29. [38]

    Halle- switch: Controlling object hallucination in large vision language models

    Bohan Zhai, Shijia Yang, Chenfeng Xu, Sheng Shen, Kurt Keutzer, and Manling Li. Halle- switch: Controlling object hallucination in large vision language models. arXiv e-prints , (arXiv-2310), 2023. 4

    2023

  30. [39]

    Mul- timodal chain-of-thought reasoning in language models

    Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Mul- timodal chain-of-thought reasoning in language models. arXiv preprint, (2302.00923), 2023. 3

    2023 arXiv

  31. [40]

    Mitigating object hallucination in large vision-language models via classifier-free guidance

    Linxi Zhao, Yihe Deng, Weitong Zhang, and Quanquan Gu. Mitigating object hallucination in large vision-language models via classifier-free guidance. arXiv preprint, (2402.08680), 2024. 4 14

    2024 arXiv

  32. [41]

    Beyond hallucinations:enhancing lvlms through hallucination-aware direct preference optimization

    Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations:enhancing lvlms through hallucination-aware direct preference optimization. arXiv preprint, (2311.16839), 2023. 4

    2023 arXiv

  33. [42]

    Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models

    Ge Zheng, Bin Yang, Jiajin Tang, Hong-yu Zhou, and Sibei Yang. Ddcot: Duty-distinct chain-of-thought prompting for multimodal reasoning in language models. NeurlPS, 2023. 4

    2023

  34. [43]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592, 2023. 9 15 A Training Parameters Detail Pre-training We directly use the pretrained w...

    2023 arXiv

  35. [44]

    MME offers a comprehesive assessment for different aspects of a MLLM’s ability including perception (coarse-grained and fine-grained object recognition and OCR) and cognition (common- sense reasoning, numerical calculation, text translation, and code reasoning), up to totally ...

  36. [45]

    All instruction-answer pairs are manually constructed and great proportion of images are newly collected in order to avoid data leakage

  37. [46]

    The unfair advantage of prompt engineering is avoided

    The instructions are made concise so as to be similar to commonly used ones. The unfair advantage of prompt engineering is avoided

  38. [47]

    yes" or

    The answers are simple "yes" or "no", which is accurate, objective and convenient for quantitative analysis. Hence, MME is an accurate, objective, fair and comprehensive benchmark for MLLM’s visual perception and cognition capabilities. C.3 TextVQA TextVQA dataset, introduced ...

    2014

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.