WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata

Abhishek Dutta; Andrew Zisserman; David M. S. Pinto; Horace Lee; Prasanna Sridhar

REVIEW 3 major objections 4 minor 15 references

This paper presents WISE, an open-source engine that unifies scene, object, face, audio-event, speech, and metadata search in one tool for image and video collections.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 23:40 UTC pith:4PQFCDQC

load-bearing objection Useful open-source systems paper that extends WISE to audio/face/speech/composite search, but the lack of any retrieval-accuracy evaluation leaves the central search-utility claim unverified. the 3 major comments →

arxiv 2602.12819 v2 pith:4PQFCDQC submitted 2026-02-13 cs.IR cs.CV

WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata

Prasanna Sridhar , Horace Lee , David M. S. Pinto , Andrew Zisserman , Abhishek Dutta This is my paper

classification cs.IR cs.CV

keywords multimodal retrievalaudiovisual searchvideo searchface searchspeech retrievalaudio event detectionvector searchopen-source software

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces WISE, an open-source search engine that brings together six retrieval modes in one tool: scene-level visual search by text or image, object-level detection with localisation, face matching, audio-event retrieval, speech transcription search, and metadata filtering. The paper's central claim is that a single practical system can integrate these modes through a modular pipeline of pretrained encoders and approximate-nearest-neighbour indexes, and that this design scales to millions of images and thousands of hours of video. A sympathetic reader would care because it makes content-based retrieval accessible to journalists, film scholars, and conservationists without machine-learning expertise, and because it can run locally on sensitive or proprietary collections. The engine also supports composite queries, such as searching for a face within a scene or filtering an object search by metadata.

Core claim

WISE's core discovery is that diverse retrieval tasks—scene, object, face, audio event, speech, metadata—can be handled by one architecture that treats every modality as feature vectors in a shared embedding space. Visual frames are embedded at scene level and region level; audio windows are embedded for acoustic events; speech is transcribed to text; and metadata is indexed with full-text search. All embeddings are stored in a vector store supporting approximate nearest-neighbour retrieval, and the same encoders are applied to queries at search time. Composite queries are implemented by combining filters across these indexes. The result is a system that performs content-based retrieval at s

What carries the argument

The central mechanism is the two-stream processing pipeline: a visual stream of images and sampled video frames, and an audio stream of windowed snippets. Pre-trained feature extractors map each stream into vectors—scene-level embeddings for whole frames, region-level embeddings for objects and faces, and acoustic embeddings for audio windows—while an automatic-speech-recognition stage transcribes spoken words into text. These vectors are stored in approximate-nearest-neighbour indexes (with an inverted-file index, optionally product-quantised for region features), and media-level metadata is indexed with full-text search. At query time, the query is embedded with the same extractor, and nea

Load-bearing premise

The system's usefulness rests on the assumption that off-the-shelf pretrained encoders embed archival, historical, and wildlife content accurately enough that nearest-neighbour search returns relevant results; the paper does not measure this retrieval accuracy.

What would settle it

Run a benchmark that compares WISE's retrieval precision and recall against human-annotated relevance judgments on a diverse set of collections (e.g., historical films, wildlife footage, news archives). If recall@k on these target domains is close to random or far below the paper's implied usefulness, the claim that WISE provides practical multimodal search would be falsified.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

Non-technical users can search private or sensitive audiovisual collections locally, without uploading content to third-party services.
Composite queries across modalities enable new forms of exploration, such as retrieving 'trains in Germany' or 'this actor in a war scene' from historical archives.
The modular design allows new feature extractors to be swapped in, so the same engine can be repurposed for other domains, such as studying chimpanzee behaviour from camera-trap footage.
With aggregator mode, a collection can be split across multiple machines and queried as a single unified index, supporting deployments with millions of images or thousands of hours of video.
Indexing happens once offline, and retrieval returns results in under a second on large datasets, making the system practical for daily use.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Because the paper reports no retrieval-accuracy numbers on the target domains, the practical value will hinge on how well the chosen pretrained encoders generalise to archival, historical, and wildlife content; the system is best thought of as a framework whose quality tracks its embeddings.
The same architecture could naturally support cross-modal queries that are not currently exposed, such as retrieving a video segment by simultaneous audio and visual similarity, or ranking results by joint audio-visual evidence.
The engine could serve as a reproducible harness for evaluating new embedding models on real-world retrieval tasks, since swapping a single extractor changes the entire search behaviour.
If open-source deployments like this become standard, they could shift the default for media archives from metadata curation to content-based exploration, changing how researchers and journalists formulate queries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Useful open-source systems paper that extends WISE to audio/face/speech/composite search, but the lack of any retrieval-accuracy evaluation leaves the central search-utility claim unverified.

read the letter

You should know this before reading: WISE is a disciplined integration of existing feature extractors into a single queryable audiovisual search engine, with open-source code and a modular design. The genuinely new thing is the composite multimodal capability — combining face, object, scene, audio, speech, and metadata filters in one tool. The earlier WISE was image-only; this version adds audio, speech, face, and composed retrieval. That is a real system contribution, and for archivists or journalists working with local collections, it plausibly fills a gap. The paper also ships code, which is more than many systems papers do, and the architecture description is clear enough to reimplement.

The soft spots are exactly where the stress-test note lands. There is no quantitative retrieval-accuracy evaluation — no recall@k, no mAP, no comparison against baselines, no per-domain accuracy for the claimed use cases (historical archives, wildlife, journalism). The qualitative examples are nice but not evidence that the off-the-shelf models align well enough for unseen archival content. The scaling claims (55M images, 6,000 hours, sub-second latency, 10 min/hour video) are stated without hardware details or a latency measurement methodology. The paper acknowledges IVFPQ has "minor impact" on accuracy but never quantifies that impact. For a paper whose core promise is "search that works," that is a load-bearing omission — not a fatal one, since the ingredients are standard and the system clearly runs, but it prevents me from confidently recommending the claims as verified.

There is also a stray line at the top of the paper — "Wait, what would her ideal man be?" — that looks like a leftover from a figure or a template artifact. It doesn't affect the science, but it should be cleaned up before publication. The citation pattern is fine; using WhisperX and prior WISE is legitimate reuse, not circularity, since there is no benchmark defined in terms of those tools.

Who is this for? Someone building a similar retrieval pipeline or wanting a local, open-source tool for media archives will find the modular architecture and composite-query design worth studying. The paper is honest about being an integration effort; it doesn't oversell novelty. But it needs a serious referee who will push for an evaluation section: at least one domain with retrieval metrics, hardware setup, and latency profiling. I would send it to peer review with the expectation of major revision, not desk reject it.

Bottom line: the system is real and potentially valuable; the paper is fine as a systems note but not yet a verified claims paper. Engage with it, but require evidence.

Referee Report

3 major / 4 minor

Summary. This paper introduces WISE, a modular open-source audiovisual search engine. It indexes images and video through scene-level OpenCLIP embeddings, OWLv2 object proposals, InsightFace face embeddings, CLAP audio embeddings, and WhisperX ASR transcripts, with Faiss approximate nearest-neighbour indices and SQLite FTS for metadata. The paper describes composite queries (metadata + visual, image + text, face + scene), an aggregator mode for distributed search, and reports deployments on 55M images and 6,000 hours of video. The reported evidence is qualitative: illustrative retrieval examples in Figs. 1–2 and narrative case studies in §5. No retrieval-accuracy metrics, latency measurement methodology, or hardware/configuration details are provided.

Significance. WISE addresses a real gap: non-expert access to multimodal search over large private and archival collections. The modular design and open-source release are concrete strengths; integration with standard models and Faiss makes the system plausible and easy to adapt, and the composite-query interface is a useful practical contribution. If the scaling and accuracy claims were verified, the system would be valuable to journalism, archival research, and wildlife conservation. However, as submitted, the central search-utility claim is not empirically supported. The paper is a system description with qualitative demonstrations only, so its significance is conditional on a quantitative evaluation demonstrating that the embedding and ANN stack returns relevant results across modalities and target domains.

major comments (3)

[§3, §5; Figs. 1–2] No retrieval-accuracy evaluation is provided. The core claim that WISE is a useful multimodal search engine is supported only by qualitative examples and anecdotal case studies. This leaves the central capability unverified, especially because the system composes off-the-shelf embeddings (OpenCLIP, OWLv2, InsightFace, CLAP, WhisperX) whose cross-modal alignment on archival, historical, and wildlife collections is an empirical assumption. Please add quantitative retrieval evaluation per modality and for composite queries, e.g. recall@k or mAP on standard benchmarks, together with exact index parameters. Without this, the paper cannot substantiate that the system retrieves relevant content rather than merely performing nearest-neighbour search.
[§4, High-performance] The performance claims are not measurable: 'A one hour video can be processed in under 10 minutes on a modern computer with a GPU' and 'WISE returns results in under 1 second' are stated without hardware configuration, dataset size, index types, nlist/nprobe or IVFPQ settings, query batch, or measurement methodology. Since the choice of approximate index directly affects latency and recall, please report concrete measurements with hardware, index parameters, and percentiles on at least one large collection.
[§3, indexing pipeline] The manuscript acknowledges that the more aggressive IVFPQ compression 'can be used to reduce compute and storage costs with a minor impact on retrieval accuracy' but does not quantify this impact. Frame sampling at 2 fps and 4 s audio windows with 2 s overlap are also free parameters that affect recall, especially for short events. These choices are load-bearing for the system's search utility; please report accuracy/latency/storage trade-offs for these choices and justify the defaults used in any evaluation.

minor comments (4)

[After author list, before Fig. 1] The line 'Wait, what would her ideal man be?' appears immediately after the author list and before Figure 1. It looks like a stray caption or placeholder; remove or integrate it into the figure caption.
[§3, Fig. 3] Several encoding/formatting issues appear: 'sampled, for example, at2frames per second' is missing a space; 'S�Lite FTS' and 'Type�uery' contain replacement characters. These should be fixed in the camera-ready version.
[§3, §4] For reproducibility, please specify the exact model variants and checkpoints used (e.g., OpenCLIP architecture and pretraining data, CLAP checkpoint, WhisperX model size) and the Faiss index hyperparameters. Currently only 'e.g.' examples are given, so a reader cannot replicate the described deployments.
[§5, public demos] The public demos are referenced via a footnote URL, but no details are given about which collections, model versions, or index settings the demos use. If the demos are intended as evidence of deployment, please state what they demonstrate and for how long they have been available.

Circularity Check

0 steps flagged

No circular derivation: WISE is an integration paper whose search-utility claim is unverified but not circular.

full rationale

WISE is a systems/integration paper; it does not present a derivation from first principles or a fitted prediction. The retrieval pipeline relies on pre-trained external embeddings (OpenCLIP, OWLv2, InsightFace, CLAP, WhisperX) and Faiss indices; Section 3 states that these feature extractors map text, images, video frames, and audio into a shared vector space where semantically related items lie close together. That is an assumption about the pretrained models, not a result derived from the paper's own outputs. No parameter is fitted to a target result and then reported as a prediction, and no target result is defined in terms of the system's own embeddings. The paper cites several self-authored components (WhisperX [1], ChimpUFE [8], prior WISE workshop paper [13]), but these are cited as existing tools or extensions, not as the justification for the central claim, and they are not invoked as a uniqueness theorem or as an unverified load-bearing premise. The main weakness is the absence of a quantitative retrieval-accuracy evaluation, so the practical usefulness and scale claims are unverified; that is a correctness/evaluation gap, not circularity.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 0 invented entities

The paper introduces no new fitted parameters beyond hand-selected sampling defaults and postulates no new entities. It depends on the availability and quality of existing pretrained multimodal models and on the adequacy of fixed temporal sampling.

free parameters (2)

frame sampling rate = 2 frames per second
Chosen by hand in Section 3 as an example; affects granularity of video search but is not fitted to a benchmark.
audio window and overlap = 4 s window, 2 s overlap
Chosen by hand in Section 3 as an example; affects audio event and speech search granularity.

axioms (3)

domain assumption Pretrained feature extractors (OpenCLIP, OWLv2, InsightFace, CLAP, WhisperX) provide semantically aligned embeddings for retrieval
Section 3 relies on these models to map queries and media into a shared vector space, but no accuracy validation is provided.
domain assumption Uniform temporal sampling of frames and audio snippets captures all searchable content
Section 3 uses 2 fps frame sampling and 4s/2s audio windows; this assumes events are visible/audible within sampled segments.
domain assumption Approximate nearest neighbor search returns sufficiently accurate matches
Section 3 uses Faiss IVFFlat/IVFPQ without reporting recall or search quality on target collections.

pith-pipeline@v1.3.0-alltime-deepseek · 5518 in / 8471 out tokens · 76786 ms · 2026-08-02T23:40:58.856269+00:00 · methodology

0 comments

read the original abstract

In this paper, we present WISE, an open-source audiovisual search engine which integrates a range of multimodal retrieval capabilities into a single, practical tool accessible to users without machine learning expertise. WISE supports natural-language and reverse-image queries at both the scene level (e.g. empty street) and object level (e.g. horse) across images and videos; face-based search for specific individuals; audio retrieval of acoustic events using text (e.g. wood creak) or an audio file; search over automatically transcribed speech; and filtering by user-provided metadata. Rich insights can be obtained by combining queries across modalities -- for example, retrieving German trains from a historical archive by applying the object query "train" and the metadata query "Germany", or searching for a face in a place. By employing vector search techniques, WISE can scale to support efficient retrieval over millions of images or thousands of hours of video. Its modular architecture facilitates the integration of new models. WISE can be deployed locally for private or sensitive collections, and has been applied to various real-world use cases. Our code is open-source and available at https://gitlab.com/vgg/wise/wise.

Figures

Figures reproduced from arXiv: 2602.12819 by Abhishek Dutta, Andrew Zisserman, David M. S. Pinto, Horace Lee, Prasanna Sridhar.

**Figure 1.** Figure 1: WISE enables search across visual, audio, and metadata streams. Visual search operates at both the scene level ( [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

**Figure 2.** Figure 2: WISE supports multimodal search which allows [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** Figure 3: WISE organises the media library into two streams: visual and audio. Scene and region level features (or embeddings) [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

**Figure 4.** Figure 4: WISE can operate in aggregator mode, where a large [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: WISE has proved invaluable for journalists inves [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗

discussion (0)

Reference graph

Works this paper leans on

15 extracted references · 1 canonical work pages

[1]

Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. InInterspeech 2023. 4489–4493. doi:10.21437/Interspeech.2023-78

doi:10.21437/interspeech.2023-78 2023
[2]

Max Bain, Arsha Nagrani, Daniel Schofield, Sophie Berdugo, Joana Bessa, Jake Owen, Kimberley J Hockings, Tetsuro Matsuzawa, Misato Hayashi, Dora Biro, et al. 2021. Automated audiovisual behavior recognition in wild primates.Science advances7, 46 (2021), eabi4883

2021
[3]

Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. 2019. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. InCVPR

2019
[4]

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG]

Pith/arXiv arXiv 2024
[5]

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang
[6]

Richard Hipp

D. Richard Hipp. 2025.SQLite. https://sqlite.org/

2025
[7]

Chuong Huynh, Jinyu Yang, Ashish Tawari, Mubarak Shah, Son Tran, Raffay Hamid, Trishul Chilimbi, and Abhinav Shrivastava. 2025. Collm: A large language model for composed image retrieval. InProceedings of the Computer Vision and Pattern Recognition Conference. 3994–4004

2025
[8]

Vladimir Iashin, Horace Lee, Dan Schofield, and Andrew Zisserman. 2025. Self- supervised Learning on Camera Trap Footage Yields a Strong Universal Face Embedder.arXiv preprint arXiv:2507.10552(2025)

Pith/arXiv arXiv 2025
[9]

2025.OpenCLIP

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John 4 WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2025.OpenCLIP. doi:10.5281/zenodo.17171361

work page doi:10.5281/zenodo.17171361 2025
[10]

Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin’ichi Satoh. 2018. A survey of product quantization.ITE Transactions on Media Technology and Applications 6, 1 (2018), 2–10

2018
[11]

Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. 2023. Scaling open- vocabulary object detection.Advances in Neural Information Processing Systems 36 (2023), 72983–73007

2023
[12]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al
[13]

Prasanna Sridhar, Horace Lee, Abhishek Dutta, and Andrew Zisserman. 2023. WISE image search engine (WISE). InWiki workshop, virtual event. Received 12 February 2026 5

2023
[2021]

In International conference on machine learning

Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763
[2023]

InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

2023

[1] [1]

Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. InInterspeech 2023. 4489–4493. doi:10.21437/Interspeech.2023-78

doi:10.21437/interspeech.2023-78 2023

[2] [2]

Max Bain, Arsha Nagrani, Daniel Schofield, Sophie Berdugo, Joana Bessa, Jake Owen, Kimberley J Hockings, Tetsuro Matsuzawa, Misato Hayashi, Dora Biro, et al. 2021. Automated audiovisual behavior recognition in wild primates.Science advances7, 46 (2021), eabi4883

2021

[3] [3]

Jiankang Deng, Jia Guo, Xue Niannan, and Stefanos Zafeiriou. 2019. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. InCVPR

2019

[4] [4]

Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre-Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. 2024. The Faiss library. (2024). arXiv:2401.08281 [cs.LG]

Pith/arXiv arXiv 2024

[5] [5]

Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang

[6] [6]

Richard Hipp

D. Richard Hipp. 2025.SQLite. https://sqlite.org/

2025

[7] [7]

Chuong Huynh, Jinyu Yang, Ashish Tawari, Mubarak Shah, Son Tran, Raffay Hamid, Trishul Chilimbi, and Abhinav Shrivastava. 2025. Collm: A large language model for composed image retrieval. InProceedings of the Computer Vision and Pattern Recognition Conference. 3994–4004

2025

[8] [8]

Vladimir Iashin, Horace Lee, Dan Schofield, and Andrew Zisserman. 2025. Self- supervised Learning on Camera Trap Footage Yields a Strong Universal Face Embedder.arXiv preprint arXiv:2507.10552(2025)

Pith/arXiv arXiv 2025

[9] [9]

2025.OpenCLIP

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John 4 WISE: A Multimodal Search Engine for Visual Scenes, Audio, Objects, Faces, Speech, and Metadata Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. 2025.OpenCLIP. doi:10.5281/zenodo.17171361

work page doi:10.5281/zenodo.17171361 2025

[10] [10]

Yusuke Matsui, Yusuke Uchida, Hervé Jégou, and Shin’ichi Satoh. 2018. A survey of product quantization.ITE Transactions on Media Technology and Applications 6, 1 (2018), 2–10

2018

[11] [11]

Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. 2023. Scaling open- vocabulary object detection.Advances in Neural Information Processing Systems 36 (2023), 72983–73007

2023

[12] [12]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sand- hini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al

[13] [13]

Prasanna Sridhar, Horace Lee, Abhishek Dutta, and Andrew Zisserman. 2023. WISE image search engine (WISE). InWiki workshop, virtual event. Received 12 February 2026 5

2023

[14] [2021]

In International conference on machine learning

Learning transferable visual models from natural language supervision. In International conference on machine learning. PmLR, 8748–8763

[15] [2023]

InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Clap learning audio concepts from natural language supervision. InICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1–5

2023