REVIEW 6 cited by
CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
When exploring the development of Artificial General Intelligence (AGI), a critical task for these models involves interpreting and processing information from multiple image inputs. However, Large Multimodal Models (LMMs) encounter two issues in such scenarios: (1) a lack of fine-grained perception, and (2) a tendency to blend information across multiple images. We first extensively investigate the capability of LMMs to perceive fine-grained visual details when dealing with multiple input images. The research focuses on two aspects: first, image-to-image matching (to evaluate whether LMMs can effectively reason and pair relevant images), and second, multi-image-to-text matching (to assess whether LMMs can accurately capture and summarize detailed image information). We conduct evaluations on a range of both open-source and closed-source large models, including GPT-4V, Gemini, OpenFlamingo, and MMICL. To enhance model performance, we further develop a Contrastive Chain-of-Thought (CoCoT) prompting approach based on multi-input multimodal models. This method requires LMMs to compare the similarities and differences among multiple image inputs, and then guide the models to answer detailed questions about multi-image inputs based on the identified similarities and differences. Our experimental results showcase CoCoT's proficiency in enhancing the multi-image comprehension capabilities of large multimodal models.
Forward citations
Cited by 6 Pith papers
-
PDB-Eval: An Evaluation of Large Multimodal Models for Description and Explanation of Personalized Driving Behavior
Introduces PDB-Eval, a dual-view benchmark for fine-grained driver behavior description and explanation, and shows fine-tuning on it boosts performance on driving QA and downstream intention and recognition tasks.
-
MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
MIND improves multimodal reasoning by training on diverse correct and deliberately wrong rationales with two-stage correction and contrastive alignment, reporting SOTA on ScienceQA, A-OKVQA, and M3CoT.
-
VisRAG2.0: Mitigating Visual Hallucinations via Evidence-Guided Multi-Image Reasoning in Visual Retrieval-Augmented Generation
EVisRAG, an evidence-guided observe-record-reason-answer pipeline trained with reward-scoped GRPO, improves multi-image visual QA accuracy by about 19% over its backbone VLM across five benchmarks.
-
From Answers to Rationales: Self-Aligning Multimodal Reasoning with Answer-Oriented Chain-of-Thought
Answer-oriented chain-of-thought prompts that generate both positive and negative reasoning data, combined with iterative DPO, improve multimodal LLM reasoning on several benchmarks.
-
Extracting Multimodal Learngene in CLIP: Unveiling the Multimodal Generalizable Knowledge
MM-LG extracts a compact multimodal and unimodal block set from CLIP via distillation and uses it to initialize smaller vision-language and vision models, outperforming previous Learngene methods and sometimes pre-tra...
-
WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning
WeThink, a 120K-image QA dataset with AI-generated reasoning paths, combined with hybrid-reward reinforcement learning, improves a 7B vision-language model across 14 benchmarks.
Discussion (0). Sign in to comment.