Pith. sign in

REVIEW 9 cited by

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.04292 v1 pith:FOZAVMSD submitted 2024-06-06 cs.IR cs.CLcs.CV

Junjie Zhou , Zheng Liu , Shitao Xiao , Bo Zhao , Yongping Xiong This is my paper
classification cs.IRcs.CLcs.CV
keywords datamulti-modalembeddingretrievalcapabilitymodeltextvista
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Multi-modal retrieval becomes increasingly popular in practice. However, the existing retrievers are mostly text-oriented, which lack the capability to process visual information. Despite the presence of vision-language models like CLIP, the current methods are severely limited in representing the text-only and image-only data. In this work, we present a new embedding model VISTA for universal multi-modal retrieval. Our work brings forth threefold technical contributions. Firstly, we introduce a flexible architecture which extends a powerful text encoder with the image understanding capability by introducing visual token embeddings. Secondly, we develop two data generation strategies, which bring high-quality composed image-text to facilitate the training of the embedding model. Thirdly, we introduce a multi-stage training algorithm, which first aligns the visual token embedding with the text encoder using massive weakly labeled data, and then develops multi-modal representation capability using the generated composed image-text data. In our experiments, VISTA achieves superior performances across a variety of multi-modal retrieval tasks in both zero-shot and supervised settings. Our model, data, and source code are available at https://github.com/FlagOpen/FlagEmbedding.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Purifying Multimodal Retrieval: Fragment-Level Evidence Selection for RAG

    cs.IR 2026-04 unverdicted novelty 7.0 of 10

    FES-RAG reframes multimodal RAG as fragment-level selection using Fragment Information Gain to outperform document-level methods with up to 27% relative CIDEr gains on M2RAG while shortening context.

  2. Adapting MLLMs for Nuanced Video Retrieval

    cs.CV 2025-12 unverdicted novelty 7.0 of 10

    Text-only contrastive fine-tuning of an MLLM with hard negatives produces embeddings that handle temporal, negation, and multimodal nuances in video retrieval and achieves SOTA performance.

  3. UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    cs.IR 2026-08 conditional novelty 6.0 of 10

    A lightweight reranker that uses coarse retrieval descriptors and retrieval-source-conditioned modality gating improves heterogeneous-source entity retrieval and downstream VQA accuracy on E-VQA and InfoSeek.

  4. UniHEAR: Unified Heterogeneous-Source Attentive Retrieval for Knowledge-Based Visual Question Answering

    cs.IR 2026-08 conditional novelty 6.0 of 10

    UniHEAR combines image-to-image and image-to-text candidate retrieval with source-aware attention reranking, improving Recall@1 over prior reranking methods on E-VQA and InfoSeek.

  5. Language-Instructed Vision Embeddings for Controllable and Generalizable Perception

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    LIVE uses language to generate task-centric vision embeddings at inference, reducing hallucinations by 34 points on MMVP, outperforming larger VLMs on VQA, and generalizing to unseen tasks.

  6. CausalEmbed: Auto-Regressive Multi-Vector Generation in Latent Space for Visual Document Embedding

    cs.CL 2026-01 unverdicted novelty 6.0 of 10

    CausalEmbed uses auto-regressive generation with iterative margin loss to produce multi-vector embeddings that reduce visual token counts 30-155x while retaining competitive performance on VDR benchmarks.

  7. E5-V: Universal Embeddings with Multimodal Large Language Models

    cs.CL 2024-07 unverdicted novelty 6.0 of 10

    E5-V produces strong universal multimodal embeddings from MLLMs trained solely on text pairs, often surpassing prior methods across retrieval and related tasks without multimodal fine-tuning.

  8. Reconstructing Content with Collaborative Attention for Universal Multimodal Representation Learning

    cs.IR 2026-03 conditional novelty 5.0 of 10

    CoCoA forces an MLLM to reconstruct masked text through a single EOS token, improving multimodal embedding quality on MMEB-V1 and matching MoCa at 3B with far less pretraining data.

  9. Unveil: Unified Visual-Textual Integration and Distillation for Multi-modal Document Retrieval

    cs.CL 2026-05 unverdicted novelty 4.0 of 10

    Unveil proposes a visual-textual embedding model for multi-modal documents that is distilled into an efficient visual-only retriever.

Pith tools