{"id":"fb5d71cc-35d5-472c-9811-fbd0fb197b7c","arxiv_id":"2602.12819","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"WISE is an open-source multimodal search engine that retrieves images, video, audio, faces, speech, and metadata using text or media queries.","lead":"This paper describes WISE, an open-source search engine that can find images, videos, audio, faces, and spoken words using natural-language or media queries. It combines several pretrained multimodal models into one tool that can be run locally on private collections.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No retrieval-accuracy evaluation leaves the core search-utility claim unverified.","rationale":"The reader's weakest assumption—that off-the-shelf embeddings produce accurate retrieval in target domains—is essentially the same concern I identify. My stress-test emphasizes that the paper provides no quantitative evaluation of retrieval accuracy at all, not just for target domains. This is load-bearing because 'search engine' implies relevant results, not just fast nearest-neighbour search. The lack of latency and configuration details further weakens the scaling claims, but the retrieval-accuracy gap is the most fundamental. I agree with the CONDITIONAL verdict: the paper is a plausible system description with open-source code, but its central claims need empirical substantiation. No new concern beyond the reader's was identified, so the verdict should remain unchanged.","tokens_in":5694,"tokens_out":4380,"duration_ms":45792,"concrete_test":"Run WISE's retrieval pipeline on a standard benchmark with ground-truth relevance. For example, use the MS-COCO validation set for scene-level and object-level visual retrieval (with captions/object labels as queries) and AudioCaps or ESC-50 for audio event retrieval. Compute Recall@10 and mean Average Precision for a sample of 100–500 queries, and compare against a naive baseline such as direct CLIP/CLAP cosine similarity without approximate indexing. Also measure the recall drop when switching from IVFFlat to IVFPQ and when reducing frame sampling from 2 fps to 1 fps. If Recall@10 is below, say, 50% on these standard benchmarks, the claimed search utility is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that WISE is a functional audiovisual search engine depends on the assumption that its feature extraction and approximate nearest-neighbour pipeline retrieves relevant results. The paper provides only qualitative examples (Figs. 1–2) and no quantitative retrieval-accuracy metrics. Section 3 describes the use of off-the-shelf models (OpenCLIP, OWLv2, InsightFace, CLAP, WhisperX) and Faiss indices (IVFFlat, IVFPQ), but never evaluates how well these embeddings align for the claimed use cases (historical archives, wildlife, journalism). The scale claims in Section 4—55M images, 6,000 hours of video, sub-second latency—are similarly unsupported by measurements or a described methodology. This matters because the choice of approximate indices and frame sampling (2 fps) can materially degrade recall, and the paper explicitly acknowledges IVFPQ has an impact on accuracy but does not quantify it. Without an accuracy evaluation, the system could be a nearest-neighbour retrieval tool with no guarantee of useful search results, undermining the core claim of being a search engine.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces WISE, a modular open-source audiovisual search engine. It indexes images and video through scene-level OpenCLIP embeddings, OWLv2 object proposals, InsightFace face embeddings, CLAP audio embeddings, and WhisperX ASR transcripts, with Faiss approximate nearest-neighbour indices and SQLite FTS for metadata. The paper describes composite queries (metadata + visual, image + text, face + scene), an aggregator mode for distributed search, and reports deployments on 55M images and 6,000 hours of video. The reported evidence is qualitative: illustrative retrieval examples in Figs. 1–2 and narrative case studies in §5. No retrieval-accuracy metrics, latency measurement methodology, or hardware/configuration details are provided.","tokens_in":5939,"tokens_out":4244,"duration_ms":39892,"significance":"WISE addresses a real gap: non-expert access to multimodal search over large private and archival collections. The modular design and open-source release are concrete strengths; integration with standard models and Faiss makes the system plausible and easy to adapt, and the composite-query interface is a useful practical contribution. If the scaling and accuracy claims were verified, the system would be valuable to journalism, archival research, and wildlife conservation. However, as submitted, the central search-utility claim is not empirically supported. The paper is a system description with qualitative demonstrations only, so its significance is conditional on a quantitative evaluation demonstrating that the embedding and ANN stack returns relevant results across modalities and target domains.","major_comments":[{"comment":"No retrieval-accuracy evaluation is provided. The core claim that WISE is a useful multimodal search engine is supported only by qualitative examples and anecdotal case studies. This leaves the central capability unverified, especially because the system composes off-the-shelf embeddings (OpenCLIP, OWLv2, InsightFace, CLAP, WhisperX) whose cross-modal alignment on archival, historical, and wildlife collections is an empirical assumption. Please add quantitative retrieval evaluation per modality and for composite queries, e.g. recall@k or mAP on standard benchmarks, together with exact index parameters. Without this, the paper cannot substantiate that the system retrieves relevant content rather than merely performing nearest-neighbour search.","section":"§3, §5; Figs. 1–2"},{"comment":"The performance claims are not measurable: 'A one hour video can be processed in under 10 minutes on a modern computer with a GPU' and 'WISE returns results in under 1 second' are stated without hardware configuration, dataset size, index types, nlist/nprobe or IVFPQ settings, query batch, or measurement methodology. Since the choice of approximate index directly affects latency and recall, please report concrete measurements with hardware, index parameters, and percentiles on at least one large collection.","section":"§4, High-performance"},{"comment":"The manuscript acknowledges that the more aggressive IVFPQ compression 'can be used to reduce compute and storage costs with a minor impact on retrieval accuracy' but does not quantify this impact. Frame sampling at 2 fps and 4 s audio windows with 2 s overlap are also free parameters that affect recall, especially for short events. These choices are load-bearing for the system's search utility; please report accuracy/latency/storage trade-offs for these choices and justify the defaults used in any evaluation.","section":"§3, indexing pipeline"}],"minor_comments":[{"comment":"The line 'Wait, what would her ideal man be?' appears immediately after the author list and before Figure 1. It looks like a stray caption or placeholder; remove or integrate it into the figure caption.","section":"After author list, before Fig. 1"},{"comment":"Several encoding/formatting issues appear: 'sampled, for example, at2frames per second' is missing a space; 'S�Lite FTS' and 'Type�uery' contain replacement characters. These should be fixed in the camera-ready version.","section":"§3, Fig. 3"},{"comment":"For reproducibility, please specify the exact model variants and checkpoints used (e.g., OpenCLIP architecture and pretraining data, CLAP checkpoint, WhisperX model size) and the Faiss index hyperparameters. Currently only 'e.g.' examples are given, so a reader cannot replicate the described deployments.","section":"§3, §4"},{"comment":"The public demos are referenced via a footnote URL, but no details are given about which collections, model versions, or index settings the demos use. If the demos are intended as evidence of deployment, please state what they demonstrate and for how long they have been available.","section":"§5, public demos"}],"recommendation":"major_revision","confidential_remarks":"This is a well-written system paper with a credible architecture, but the stress-test concern is valid: there is no quantitative evaluation of retrieval accuracy, and the performance claims are not backed by methodology or hardware details. I would support acceptance after a major revision that adds per-modality and composite retrieval metrics, latency measurements with configuration details, and an ablated justification of the sampling and index choices. Circularity is not a concern here: the reuse of the authors' own components is disclosed and is not used as an evaluation criterion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this before reading: WISE is a disciplined integration of existing feature extractors into a single queryable audiovisual search engine, with open-source code and a modular design. The genuinely new thing is the composite multimodal capability — combining face, object, scene, audio, speech, and metadata filters in one tool. The earlier WISE was image-only; this version adds audio, speech, face, and composed retrieval. That is a real system contribution, and for archivists or journalists working with local collections, it plausibly fills a gap. The paper also ships code, which is more than many systems papers do, and the architecture description is clear enough to reimplement.\n\nThe soft spots are exactly where the stress-test note lands. There is no quantitative retrieval-accuracy evaluation — no recall@k, no mAP, no comparison against baselines, no per-domain accuracy for the claimed use cases (historical archives, wildlife, journalism). The qualitative examples are nice but not evidence that the off-the-shelf models align well enough for unseen archival content. The scaling claims (55M images, 6,000 hours, sub-second latency, 10 min/hour video) are stated without hardware details or a latency measurement methodology. The paper acknowledges IVFPQ has \"minor impact\" on accuracy but never quantifies that impact. For a paper whose core promise is \"search that works,\" that is a load-bearing omission — not a fatal one, since the ingredients are standard and the system clearly runs, but it prevents me from confidently recommending the claims as verified.\n\nThere is also a stray line at the top of the paper — \"Wait, what would her ideal man be?\" — that looks like a leftover from a figure or a template artifact. It doesn't affect the science, but it should be cleaned up before publication. The citation pattern is fine; using WhisperX and prior WISE is legitimate reuse, not circularity, since there is no benchmark defined in terms of those tools.\n\nWho is this for? Someone building a similar retrieval pipeline or wanting a local, open-source tool for media archives will find the modular architecture and composite-query design worth studying. The paper is honest about being an integration effort; it doesn't oversell novelty. But it needs a serious referee who will push for an evaluation section: at least one domain with retrieval metrics, hardware setup, and latency profiling. I would send it to peer review with the expectation of major revision, not desk reject it.\n\nBottom line: the system is real and potentially valuable; the paper is fine as a systems note but not yet a verified claims paper. Engage with it, but require evidence.","headline":"Useful open-source systems paper that extends WISE to audio/face/speech/composite search, but the lack of any retrieval-accuracy evaluation leaves the central search-utility claim unverified.","tokens_in":6376,"tokens_out":1594,"would_cite":true,"duration_ms":18989,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents WISE, an open-source engine that unifies scene, object, face, audio-event, speech, and metadata search in one tool for image and video collections.","keywords":["multimodal retrieval","audiovisual search","video search","face search","speech retrieval","audio event detection","vector search","open-source software"],"falsifier":"Run a benchmark that compares WISE's retrieval precision and recall against human-annotated relevance judgments on a diverse set of collections (e.g., historical films, wildlife footage, news archives). If recall@k on these target domains is close to random or far below the paper's implied usefulness, the claim that WISE provides practical multimodal search would be falsified.","tokens_in":5629,"feed_emoji":"🔎","tokens_out":4557,"duration_ms":37608,"temperature":0.7,"pith_summary":"This paper introduces WISE, an open-source search engine that brings together six retrieval modes in one tool: scene-level visual search by text or image, object-level detection with localisation, face matching, audio-event retrieval, speech transcription search, and metadata filtering. The paper's central claim is that a single practical system can integrate these modes through a modular pipeline of pretrained encoders and approximate-nearest-neighbour indexes, and that this design scales to millions of images and thousands of hours of video. A sympathetic reader would care because it makes content-based retrieval accessible to journalists, film scholars, and conservationists without machine-learning expertise, and because it can run locally on sensitive or proprietary collections. The engine also supports composite queries, such as searching for a face within a scene or filtering an object search by metadata.","feed_headline":"WISE searches video by scene, face, sound, and spoken word","feed_subtitle":"An open-source tool brings multimodal search to private archives without machine-learning expertise","key_machinery":"The central mechanism is the two-stream processing pipeline: a visual stream of images and sampled video frames, and an audio stream of windowed snippets. Pre-trained feature extractors map each stream into vectors—scene-level embeddings for whole frames, region-level embeddings for objects and faces, and acoustic embeddings for audio windows—while an automatic-speech-recognition stage transcribes spoken words into text. These vectors are stored in approximate-nearest-neighbour indexes (with an inverted-file index, optionally product-quantised for region features), and media-level metadata is indexed with full-text search. At query time, the query is embedded with the same extractor, and nea","core_discovery":"WISE's core discovery is that diverse retrieval tasks—scene, object, face, audio event, speech, metadata—can be handled by one architecture that treats every modality as feature vectors in a shared embedding space. Visual frames are embedded at scene level and region level; audio windows are embedded for acoustic events; speech is transcribed to text; and metadata is indexed with full-text search. All embeddings are stored in a vector store supporting approximate nearest-neighbour retrieval, and the same encoders are applied to queries at search time. Composite queries are implemented by combining filters across these indexes. The result is a system that performs content-based retrieval at s","pith_inferences":["Because the paper reports no retrieval-accuracy numbers on the target domains, the practical value will hinge on how well the chosen pretrained encoders generalise to archival, historical, and wildlife content; the system is best thought of as a framework whose quality tracks its embeddings.","The same architecture could naturally support cross-modal queries that are not currently exposed, such as retrieving a video segment by simultaneous audio and visual similarity, or ranking results by joint audio-visual evidence.","The engine could serve as a reproducible harness for evaluating new embedding models on real-world retrieval tasks, since swapping a single extractor changes the entire search behaviour.","If open-source deployments like this become standard, they could shift the default for media archives from metadata curation to content-based exploration, changing how researchers and journalists formulate queries."],"forward_implications":["Non-technical users can search private or sensitive audiovisual collections locally, without uploading content to third-party services.","Composite queries across modalities enable new forms of exploration, such as retrieving 'trains in Germany' or 'this actor in a war scene' from historical archives.","The modular design allows new feature extractors to be swapped in, so the same engine can be repurposed for other domains, such as studying chimpanzee behaviour from camera-trap footage.","With aggregator mode, a collection can be split across multiple machines and queried as a single unified index, supporting deployments with millions of images or thousands of hours of video.","Indexing happens once offline, and retrieval returns results in under a second on large datasets, making the system practical for daily use."],"fun_headline_variants":["Open-source WISE finds scenes, faces, sounds, and speech","Search video by scene, face, audio, and transcript with WISE","WISE unifies multimodal search: scenes, faces, audio, speech","One engine for visual, audio, and speech search in archives"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The system's usefulness rests on the assumption that off-the-shelf pretrained encoders embed archival, historical, and wildlife content accurately enough that nearest-neighbour search returns relevant results; the paper does not measure this retrieval accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Open-source WISE finds scenes, faces, sounds, and speech","Search video by scene, face, audio, and transcript with WISE","WISE unifies multimodal search: scenes, faces, audio, speech","One engine for visual, audio, and speech search in archives"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1493,"prompt_tokens":729,"completion_tokens":764,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":688}},"tokens_in":473,"tokens_out":764,"duration_ms":6285,"temperature":1.0,"reasoning_tokens":688,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T23:40:58.856269+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a benchmark that compares WISE's retrieval precision and recall against human-annotated relevance judgments on a diverse set of collections (e.g., historical films, wildlife footage, news archives). If recall@k on these target domains is close to random or far below the paper's implied usefulness, the claim that WISE provides practical multimodal search would be falsified.","supporting_citations":[],"review_version":1}