Pith. sign in

REVIEW 11 cited by

EchoSight: Advancing Visual-Language Models with Wiki Knowledge

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2407.12735 v4 pith:YL5FFR52 submitted 2024-07-17 cs.CV

Yibin Yan , Weidi Xie This is my paper · ORCID
classification cs.CV
keywords knowledgeechosightencyclopedicmodelsansweringarticlesinfoseekintegration
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Knowledge-based Visual Question Answering (KVQA) tasks require answering questions about images using extensive background knowledge. Despite significant advancements, generative models often struggle with these tasks due to the limited integration of external knowledge. In this paper, we introduce EchoSight, a novel multimodal Retrieval-Augmented Generation (RAG) framework that enables large language models (LLMs) to answer visual questions requiring fine-grained encyclopedic knowledge. To strive for high-performing retrieval, EchoSight first searches wiki articles by using visual-only information, subsequently, these candidate articles are further reranked according to their relevance to the combined text-image query. This approach significantly improves the integration of multimodal knowledge, leading to enhanced retrieval outcomes and more accurate VQA responses. Our experimental results on the Encyclopedic VQA and InfoSeek datasets demonstrate that EchoSight establishes new state-of-the-art results in knowledge-based VQA, achieving an accuracy of 41.8% on Encyclopedic VQA and 31.3% on InfoSeek.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 11 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    WikiCLIP reaches 28.5% OVEN-unseen accuracy (vs 24.5% AutoVER) at 14.5 ms latency by vision-guided LLM embeddings plus hard-negative text swaps.

  2. UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    cs.IR 2026-08 conditional novelty 6.0 of 10

    UniHEAR combines image-to-image and image-to-text candidate retrieval with source-aware attention reranking, improving Recall@1 over prior reranking methods on E-VQA and InfoSeek.

  3. Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Controllable retrieval-difficulty curriculum plus reward-propagation sampling lets RL close the pretrain-to-KB-VQA gap and beat prior SOTA on two hard encyclopedic VQA benchmarks.

  4. Spa-VLM: Stealthy Poisoning Attacks on RAG-based VLM

    cs.CR 2025-05 conditional novelty 6.0 of 10

    Spa-VLM crafts paired adversarial images and misleading texts to poison RAG-based VLM knowledge bases, reaching attack success rates above 0.8 with just five injected entries.

  5. mR$^2$AG: Multimodal Retrieval-Reflection-Augmented Generation for Knowledge-Based VQA

    cs.AI 2024-11 conditional novelty 6.0 of 10

    mR2AG adds two reflection steps to multimodal LLMs, adaptive retrieval and evidence relevance scoring, yielding SOTA results on INFOSEEK and Encyclopedic-VQA.

  6. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  7. Augmented Vision-Language Models: A Systematic Review

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.

  8. Towards General Continuous Memory for Vision-Language Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A vision-language model can act as its own continuous memory encoder, compressing external multimodal knowledge into eight embeddings that improve reasoning when prepended to the frozen model.

  9. Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

    cs.CV 2024-11 conditional novelty 5.0 of 10

    A multimodal language model with special tokens for deciding when to retrieve and which passages to trust beats earlier retrieval-augmented methods on knowledge-based visual question answering.

  10. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

  11. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools