REVIEW 6 cited by
Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
The advancement of Large Language Models (LLMs) has brought substantial attention to the Chain of Thought (CoT) approach, primarily due to its ability to enhance the capability of LLMs on complex reasoning tasks. Moreover, the significance of CoT approaches extends to the application of LLMs for multi-modal tasks. However, the selection of optimal CoT demonstration examples in multi-modal reasoning remains less explored for LLMs due to the inherent complexity of multi-modal examples. In this paper, we introduce a novel approach that addresses this challenge by using retrieval mechanisms to dynamically and automatically select demonstration examples based on cross-modal and intra-modal similarities. Furthermore, we employ a Stratified Sampling method of categorising demonstration examples into groups based on their types and then retrieving examples from different groups respectively to promote the diversity of demonstration examples. Through a series of experiments on two popular benchmark datasets: ScienceQA and MathVista, we demonstrate that our approach significantly improves the performance of GPT-4 by 6% on ScienceQA and 12.9% on MathVista, and enhances the performance of GPT-4V on two datasets by 2.7%, substantially improving the performance of the most advanced LLMs and LMMs for complex multi-modal reasoning tasks.
Forward citations
Cited by 6 Pith papers
-
Progressive Multimodal Reasoning via Active Retrieval
AR-MCTS retrieves per-step multimodal knowledge during tree search and trains a process reward model, improving MLLM reasoning accuracy on three benchmarks by a few points.
-
CoMT: A Novel Benchmark for Chain of Multi-modal Thought on Large Vision-Language Models
CoMT is the first benchmark to ask LVLMs to produce interleaved image and text rationales, and current models perform near random on it.
-
MIND: Multi-rationale INtegrated Discriminative Reasoning Framework for Multi-modal Large Models
MIND improves multimodal reasoning by training on diverse correct and deliberately wrong rationales with two-stage correction and contrastive alignment, reporting SOTA on ScienceQA, A-OKVQA, and M3CoT.
-
FaithfulRAG: Fact-Level Conflict Modeling for Context-Faithful Retrieval-Augmented Generation
FaithfulRAG resolves knowledge conflicts in RAG by extracting the model's parametric facts, aligning them with context, and reasoning through discrepancies before generating an answer.
-
Improving Multimodal LLMs Ability In Geometry Problem Solving, Reasoning, And Multistep Scoring
GPSM4K is a new geometry QA dataset with step-by-step solutions; the paper reports that fine-tuning on it improves LVLM scores, with captioning and RAG giving smaller gains.
-
An Overview and Discussion on Using Large Language Models for Implementation Generation of Solutions to Open-Ended Problems
A survey and position paper that reviews LLM prompting, RAG, and RL techniques and argues they could support open-ended implementation generation, without presenting new results.
Discussion (0). Continue with ORCID to comment.