Pith. sign in

REVIEW 15 cited by

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2302.11713 v5 pith:LVK7KCOH submitted 2023-02-23 cs.CV cs.AIcs.CL

Yang Chen , Hexiang Hu , Yi Luan , Haitian Sun , Soravit Changpinyo , Alan Ritter , Ming-Wei Chang This is my paper · ORCID
classification cs.CVcs.AIcs.CL
keywords visualmodelsansweringinformation-seekinginfoseekpre-trainedquestionsquestion
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Pre-trained vision and language models have demonstrated state-of-the-art capabilities over existing tasks involving images and texts, including visual question answering. However, it remains unclear whether these models possess the capability to answer questions that are not only querying visual content but knowledge-intensive and information-seeking. In this study, we introduce InfoSeek, a visual question answering dataset tailored for information-seeking questions that cannot be answered with only common sense knowledge. Using InfoSeek, we analyze various pre-trained visual question answering models and gain insights into their characteristics. Our findings reveal that state-of-the-art pre-trained multi-modal models (e.g., PaLI-X, BLIP2, etc.) face challenges in answering visual information-seeking questions, but fine-tuning on the InfoSeek dataset elicits models to use fine-grained knowledge that was learned during their pre-training. Furthermore, we show that accurate visual entity recognition can be used to improve performance on InfoSeek by retrieving relevant documents, showing a significant space for improvement.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 15 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WikiCLIP: An Efficient Contrastive Baseline for Open-domain Visual Entity Recognition

    cs.CV 2026-03 unverdicted novelty 7.0 of 10

    WikiCLIP reaches 28.5% OVEN-unseen accuracy (vs 24.5% AutoVER) at 14.5 ms latency by vision-guided LLM embeddings plus hard-negative text swaps.

  2. Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video-DeepResearch trains video agents with a two-stage SFT plus GRPO pipeline to do visual web searches before text searches, and introduces a 200-question benchmark requiring both video grounding and open-web knowledge.

  3. UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    cs.IR 2026-08 conditional novelty 6.0 of 10

    UniHEAR combines image-to-image and image-to-text candidate retrieval with source-aware attention reranking, improving Recall@1 over prior reranking methods on E-VQA and InfoSeek.

  4. Wiki-R1: Incentivizing Multimodal Reasoning for Knowledge-based VQA via Data and Sampling Curriculum

    cs.CV 2026-03 conditional novelty 6.0 of 10

    Controllable retrieval-difficulty curriculum plus reward-propagation sampling lets RL close the pretrain-to-KB-VQA gap and beat prior SOTA on two hard encyclopedic VQA benchmarks.

  5. R3G: A Reasoning-Retrieval-Reranking Framework for Vision-Centric Answer Generation

    cs.CV 2026-01 unverdicted novelty 6.0 of 10

    R3G improves vision-centric VQA by generating a reasoning plan before retrieval and reranking candidate images with an MLLM judge on relevance, target match, and answerability.

  6. MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    A new one-million-sample multimodal agent tuning dataset with GPT-4o-generated rationales, reflection, and tool/RAG calls is shown to improve fine-tuned models, though training/eval benchmark overlap is not addressed.

  7. GenRecal: Generation after Recalibration from Large to Small Vision-Language Models

    cs.CL 2025-06 conditional novelty 6.0 of 10

    A learnable Recalibrator bridges different tokenizers so that small VLMs can distill knowledge from any large VLM, improving their benchmark scores.

  8. Spa-VLM: Stealthy Poisoning Attacks on RAG-based VLM

    cs.CR 2025-05 conditional novelty 6.0 of 10

    Spa-VLM crafts paired adversarial images and misleading texts to poison RAG-based VLM knowledge bases, reaching attack success rates above 0.8 with just five injected entries.

  9. UniCoRN: Unified Commented Retrieval Network with LMMs

    cs.CV 2025-02 conditional novelty 6.0 of 10

    A frozen multimodal LLM is extended with a retrieval adapter and an entity adapter to retrieve a relevant image and generate a supportive textual comment.

  10. VLR-Bench: Multilingual Benchmark Dataset for Vision-Language Retrieval Augmented Generation

    cs.CV 2024-12 conditional novelty 6.0 of 10

    The paper presents a 300-sample, three-language benchmark with two useful, two partial, and one irrelevant passage per query, together with a training set that improved passage selection in VLMs.

  11. Reason Before You Retrieve: Agentic Planning for Multi-modal RAG

    cs.AI 2026-06 reject novelty 5.0 of 10

    MM-R2 claims SOTA multimodal RAG accuracy on InfoSeek and Encyclopedic VQA via intent grounding plus a 10-topic KnowledgeMap, but its teacher trajectories leak the gold Wikipedia page and omit the image.

  12. Augmented Vision-Language Models: A Systematic Review

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A structured taxonomy of inference-time augmentation techniques that connect vision-language models to external symbolic systems, tools, and knowledge sources.

  13. MMTABREAL: Real-World Benchmark for Multimodal Table Understanding

    cs.CV 2025-05 conditional novelty 5.0 of 10

    The paper releases a 500-table, 4,021-question benchmark of real-world multimodal tables and shows that leading vision-language models drop 20-40% in accuracy relative to earlier synthetic benchmarks.

  14. Benchmarking Poisoning Attacks against Retrieval-Augmented Generation

    cs.CR 2025-05 conditional novelty 5.0 of 10

    A unified benchmark evaluation finds that existing RAG poisoning attacks remain effective on standard QA datasets, drop on expanded knowledge bases, and are only partially mitigated by current defenses.

  15. Towards General Continuous Memory for Vision-Language Models

    cs.LG 2025-05 conditional novelty 5.0 of 10

    A vision-language model can act as its own continuous memory encoder, compressing external multimodal knowledge into eight embeddings that improve reasoning when prepended to the frozen model.

Pith tools