Image search using multilingual texts: a cross-modal learning approach between image and text

Adrien Nivaggioli; Christophe Servan (LIUM); Estelle Maudet; Hicham Randrianarivo (CEDRIC); Maxime Portaz; Sylvain Peyronnet (ELM)

Image search using multilingual texts: a cross-modal learning approach between image and text

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 1903.11299 v3 pith:UIQEJWAH submitted 2019-03-27 cs.CV cs.CL

Maxime Portaz , Hicham Randrianarivo (CEDRIC) , Adrien Nivaggioli , Estelle Maudet , Christophe Servan (LIUM) , Sylvain Peyronnet (ELM) This is my paper · ORCID

classification cs.CVcs.CL

keywords imageimageslanguagesmultilingualspacetextapproachcommon

verification ladder T0 review T1 audit T2 compute T3 formal

0 comments

read the original abstract

Multilingual (or cross-lingual) embeddings represent several languages in a unique vector space. Using a common embedding space enables for a shared semantic between words from different languages. In this paper, we propose to embed images and texts into a unique distributional vector space, enabling to search images by using text queries expressing information needs related to the (visual) content of images, as well as using image similarity. Our framework forces the representation of an image to be similar to the representation of the text that describes it. Moreover, by using multilingual embeddings we ensure that words from two different languages have close descriptors and thus are attached to similar images. We provide experimental evidence of the efficiency of our approach by experimenting it on two datasets: Common Objects in COntext (COCO) [19] and Multi30K [7].

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval
cs.CV 2024-12 conditional novelty 6.0 of 10

A dynamic adapter generated from disentangled semantic and style features improves cross-lingual cross-modal retrieval over static adapters on image and video benchmarks.

Pith tools