LaVPR: Benchmarking Language and Vision for Place Recognition

Dan Badur; Ofer Idan; Yoli Shavit; Yosi Keller

REVIEW 3 major objections 6 minor 10 references

LaVPR: Benchmarking Language and Vision for Place Recognition

T0 review · 3 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Language descriptions consistently improve place recognition when visual conditions degrade, and pairing a compact vision model with text lets it rival much larger vision-only systems.

desk verdict LaVPR is a genuinely useful benchmark with a clean controlled setup, but the 'language helps under degradation' headline rests on clean-captions-plus-degraded-images with tiny test sets, so the real-world claim is ahead of the evidence. read the letter →

arxiv 2602.03253 v2 pith:R7PKWRPL submitted 2026-02-03 cs.CV

Ofer Idan , Dan Badur , Yosi Keller , Yoli Shavit This is my paper · ORCID

classification cs.CV

keywords VisualPlaceRecognitionVision-LanguageBenchmarkCross-ModalRetrievalMulti-ModalFusionLow-RankAdaptationMulti-SimilarityLossDegradedConditionsLanguage-basedLocalization

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LaVPR, a benchmark that adds over 650,000 machine-generated natural-language descriptions to established visual place recognition datasets, then uses it to test two ideas. First, fusing language with visual features makes retrieval more robust under blur, weather, and long-term change, with the largest gains on small, efficient backbones. Second, a text-only query can retrieve a place from an image database if the vision-language model is adapted with Low-Rank Adaptation and trained with a Multi-Similarity loss, giving an order-of-magnitude improvement over standard contrastive baselines. If these findings hold, language becomes a cheap and effective complement to vision for localization, especially in degraded conditions and on resource-constrained devices.

What carries the argument

LaVPR is the benchmark itself: 651,865 image-description pairs generated by a visual language model, curated through a segmentation-grounded pipeline with human-in-the-loop hallucination filtering. The two technical mechanisms are (1) late fusion for multi-modal recognition, specifically Adaptive Score Fusion with Learned Language Pooling (ADS-LLP) or simple concatenation, both trained with Multi-Similarity loss; and (2) Low-Rank Adaptation (LoRA) applied to all linear projections of a vision-language backbone, combined with Multi-Similarity loss, which aligns text and image embeddings for cross-modal retrieval.

What would settle it

Gather human-written descriptions for a sample of LaVPR queries, and apply real-world blur or weather (e.g., rain on a camera lens, motion blur from a moving vehicle) to the same places; if the language-augmented gains shrink toward zero or reverse under these conditions, the benchmark's synthetic setup does not transfer.

Watch

Extended reading notes

Core claim

The paper claims that natural-language descriptions act as a stable, high-level semantic anchor that compensates for pixel-level visual degradation in place recognition. Across several standard VPR backbones, adding text via late fusion (concatenation or adaptive score fusion with learned language pooling) yields consistent Recall@1 gains on blurred and weather-augmented subsets, with relative gains up to 162% for small supervised backbones; language-augmented compact models can match or exceed the accuracy of far larger vision-only models at lower compute. For cross-modal retrieval, the paper shows that zero-shot vision-language models and standard contrastive fine-tuning fail, but applying

Load-bearing premise

The captions were generated from clean images by a machine, and the degraded test images are synthetic, so the benchmark assumes this setup captures how real human descriptions and real degraded imagery behave in the field.

Editorial extensions

If this is right

If language-augmented fusion holds up, compact models with a small text encoder can replace much larger vision-only architectures in degraded conditions, cutting compute by over 60% in the paper's efficiency comparison.
Cross-modal retrieval with LoRA and Multi-Similarity loss provides a usable baseline for 'blind' localization from verbal descriptions, relevant to emergency response, forensic geolocation, and semantic robotics.
Language descriptions serve as a viewpoint-invariant prior: features like shop signs, building colors, and architectural details survive blur and weather that destroy pixel-level cues.
Sequential re-ranking (retrieve by vision, re-rank by text) fails because the first modality can miss the ground truth entirely; joint fusion avoids that bottleneck, guiding future system design.
The benchmark's controlled setup, preserving original geographic splits and using identical visual backbones, isolates the contribution of language, making comparisons across fusion and alignment methods fair.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not test human-written descriptions; if humans describe places differently than the VLLM does, the measured gains may change. A natural extension is collecting human descriptions for the same queries and re-running the fusion and retrieval experiments.
The degraded test sets are synthetic (Photoshop blur, generated weather overlays); real-world degraded imagery, where degradation is non-uniform and correlated with scene content, might interact differently with the language channel.
The 'language as regularizer' result suggests a testable scaling hypothesis: as visual backbones grow and become more robust, the marginal value of language shrinks, but the paper's small-backbone gains indicate that the efficient frontier of VPR may move toward multimodal compact models.
Cross-modal retrieval is still far below uni-modal performance; the paper's own limitation discussion notes this gap, implying that stronger alignment objectives or token-level interactions could be the next lever.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Read this one if you work on VPR or vision-language grounding. The contribution is real: LaVPR adds 651,865 Gemini-generated captions to five established VPR datasets, preserves the original splits, keeps backbones identical, and evaluates late fusion and cross-modal retrieval in a controlled way. The LoRA + Multi-Similarity recipe is a solid baseline, and the ablations (LoRA vs full fine-tuning, MS vs contrastive, rank and layer coverage) are informative. The efficiency result — a small ViT plus text encoder rivaling large vision-only CricaVPR on degraded subsets — is the most compelling part. The curation pipeline is also a genuine effort: segmentation grounding, VLM verification, human-in-the-loop review, and a 93.2% signage accuracy check. The dataset and code are promised. Citations look standard and appropriate.

Soft spots, in order of seriousness. First, the degraded-condition claim has a setup problem: captions are generated from clean images, and queries are synthetically degraded afterwards. So the comparison is clean text + degraded vision versus degraded vision alone. That measures whether a clean language anchor helps, not whether descriptions produced under real degradation would help. The abstract's "consistent gains in visually degraded conditions" overstates what is demonstrated. Second, the headline gains come from very small subsets: 31 queries on Amstertime-La, 86 each on MSLS-Blur/Weather, and those subsets are curated for matching scene text. No error bars or significance tests are reported. The gains may well be real — the margins are large — but the paper should show confidence intervals or a paired test and report the unselected MSLS-val numbers. Third, "consistent gains" is contradicted by Table 7: SALAD and CricaVPR are flat or negative on standard benchmarks. The claim should be scoped to degraded and text-discriminative subsets. Fourth, cross-modal "blind" localization is evaluated on captions written from the ground-truth image itself, so image-specific details are available to the text query. That is fine as a controlled probe, but not as evidence for witness or emergency-response text. Fifth, LoRA hyperparameters (rank, layers) are selected on the test set.

None of this kills the paper. The benchmark is new, large, and likely useful, and the controlled protocol is a real advance over prior language-augmented VPR work. It needs scoped claims, uncertainty quantification, and ideally a validation set with human-written captions or captions generated from degraded inputs. I would send it to peer review, and I would cite it with caveats if I worked in this area.

Referee Report

3 major / 6 minor

Summary. The paper introduces LaVPR, a benchmark that augments established VPR datasets (GSV-Cities, Pitts30K, AmsterTime, MSLS) with 651,865 VLLM-generated natural-language descriptions, together with a segmentation/VLM-based curation pipeline and a human-in-the-loop cleaning stage. Using this benchmark, the authors evaluate two paradigms: multi-modal fusion (V+L) via late fusion with frozen encoders (CAT, PA, MLP, ADS+LLP), and cross-modal retrieval (L→V) via LoRA with a Multi-Similarity loss applied to several vision-language backbones. The main empirical claims are that language gives consistent and often large relative gains on visually degraded and scene-text subsets, that language augmentation lets compact models rival larger vision-only models, and that LoRA + MS loss substantially outperforms zero-shot and standard contrastive cross-modal baselines. The dataset and code are promised for release.

Significance. If the empirical claims hold up, LaVPR would be a useful community resource: it is, to my knowledge, the first large-scale standardized benchmark of this kind for VPR, and the controlled experimental design — identical visual backbones, preserved geographic splits, detailed ablation of fusion mechanisms, and thorough documentation of the caption-generation and curation process — is a genuine strength. The cross-modal finding that LoRA combined with a retrieval-oriented MS loss provides a strong baseline across four VLMs is interesting and actionable. The release of the dataset, prompts, and pipeline details is also an important contribution. The main risk is external validity: the captions are generated from clean source images, and the most striking fusion gains come from very small, synthetically degraded subsets. The manuscript currently overstates what the controlled probe can establish about real-world deployment.

major comments (3)

[A.2.2 / Tables 5–6] The headline fusion gains on MSLS-Blur and MSLS-Weather are produced by pairing a synthetically degraded query image with a language description generated from the original clean image. The text channel therefore contains clean-view information — exact sign text, colors, storefront details — that the blur/weather augmentation removes from the visual channel. The experiment thus measures 'clean-language + degraded-vision' versus 'degraded-vision alone,' not language robustness when the input is degraded. Because the abstract and conclusion frame the result in deployment terms ('resilient to real-world stochasticity'), the paper needs at least one of: (i) captions generated from the degraded image, (ii) human-written descriptions for a subset of degraded queries, or (iii) a clear reframing of the result as a controlled probe of clean-language conditioning. As written, the external-validity
[§5.2, Table 11] The cross-modal 'blind' localization experiments use text queries produced by Gemini from the ground-truth image itself. These captions contain image-specific, viewpoint-specific details that a human witness or a captioner observing the degraded scene would not produce. No experiment uses human-written descriptions or descriptions generated from a different input or prompt. Given that the stated motivation is emergency calls and witness descriptions, the claimed feasibility of 'blind' localization remains unvalidated beyond the automatic-caption setting. I request at least a small human-written query set (even 50–100 queries) or captions generated from degraded or holdout images to establish that the task, as formulated, reflects the motivating scenario.
[Table 2 / Tables 5–7] The largest relative gains come from subsets with only 31 queries (Amstertime-La) and 86 queries (MSLS-Blur, MSLS-Weather). The paper reports no confidence intervals, significance tests, or multiple-seed variability. For n=86, a 4–6 point difference in R@1 is within the standard error, and Table 7 contains negative entries (e.g., La-CricaVPR CAT on Amstertime-La: −9.9%; MSLS-C: −3.7%). I request at least bootstrap CIs over the query set or repeated fine-tuning with different seeds to establish that the small-subset gains are not noise. This is load-bearing for the 'consistent gains' claim in the abstract.

minor comments (6)

[Abstract / Table 7] The phrase 'consistent gains' is stronger than the data show; several entries in Table 7 are negative. Suggest qualifying the claim to 'gains on most evaluated backbones and subsets, particularly small convolutional backbones and synthetically degraded conditions.'
[Table 10] The row labeled 'LaVPR (Ours)' should specify the exact method (appears to be La-MixVPR with CAT fusion) so the reader can map it to Tables 5–6.
[Tables 3 and 11, references] Typos: 'V ocabulary' in Table 3; 'EV A-CLIP-V2' in Table 11; 'Y V' in the Lai et al. reference; 'Packged' in the Xiao et al. reference.
[Table 8 caption] The 62% GFLOP reduction assumes a caption is available at query time. If the deployment setting does not provide a text description, the text-encoder cost is an additional on-demand computation; this assumption should be stated explicitly.
[A.3.1 / Table 14] The automated hallucination detector is reported to have 22% precision, and the manual HL inspection covers only the 20 largest unresolved groups rather than a random sample. The '~1% hallucination' statement should carry this sampling caveat.
[Conclusion, Limitations] The Limitations paragraph does not mention the clean-caption / degraded-image mismatch or the absence of human-written descriptions. These are central to interpreting the results and should be acknowledged.

Circularity Check

1 steps flagged · score 3.0 of 10

Degraded-subset language gains are partly built into the benchmark construction (clean captions + degraded queries); no self-citation or equation-level circularity elsewhere.

self definitional [Appendix A.2.2 (Visual Degradation Synthesis); Section 4.2 (Results & Discussion)]
"we applied synthetic transformations to the MSLS-val queries. For BLUR augmentations, we used Photoshop’s Lens Blur tool. We further utilized gemini-nano-banana to generate adverse weather conditions (e.g., heavy rain, snow, and fog) ... In these scenarios, language functions as a discriminative anchor that remains invariant to the visual domain shifts that otherwise suppress pixel-level features."

The MSLS-Blur/Weather subsets are created by degrading only the query image; the paired captions are generated by Gemini 2.5 Flash from the original clean images (Section 3.1, A.2.1). Thus the language channel is, by construction, the pre-degradation view. The paper's explanation that language 'remains invariant to the visual domain shifts' is not a learned or predicted property but an artifact of generating captions before applying blur/weather. The double-digit relative gains on these subsets (Table 7) therefore partly reduce to comparing clean-language + degraded-vision against degraded-vision alone. This is a dataset-construction confound rather than an equation-level circularity; it does not undermine the cross-modal LoRA-MS results on standard splits.

full rationale

Most of the paper is self-contained: external VPR backbones and datasets are used, captions come from an external VLLM (Gemini 2.5 Flash), retrieval metrics are standard Recall@K, and the LoRA-MS cross-modal comparisons are trained and evaluated on fixed dataset splits. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation. The one notable construction issue is the degraded-condition claim: in MSLS-Blur/Weather, captions are generated from clean images while only the query image is degraded, so the language channel is clean by construction. The paper's statement that language is 'invariant to the visual domain shifts' is therefore a property of the data-generation protocol, not an independent empirical discovery. The paper itself concedes in A.3.1 that it lacked a concrete quantitative assessment of Gemini's performance, reinforcing the external-validity caveat. This is more a dataset-validity confound (correctness risk) than a derivation-level circularity, and the cross-modal/fusion results on standard splits retain independent content. I therefore assign a low-but-nonzero score.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The ledger is clean: no new physical entities or forces. The main input assumptions are external validity of VLLM captions and synthetic degradations, plus a few hand-chosen hyperparameters. The most significant concern is that LoRA rank and target layers were selected using test-set ablations, which is a form of parameter fitting to the evaluation data.

free parameters (4)

MS loss hyperparameters (multi-modal fusion) = alpha=1, beta=50, margin=0
Given in Section A.5; chosen by hand, not fitted to test, but not justified by analysis.
MS loss hyperparameters (cross-modal retrieval) = alpha=2, beta=40, margin=0.5
Given in Section A.6; selected by the authors without ablation, affects the cross-modal results.
LoRA rank r = 64 (all linear layers)
Table 13 ablates r=16 and r=64 on the Amstertime test set and picks 64; no held-out validation split is described, so the choice is informed by test performance.
LoRA target layers = All linear layers vs. QKV
Also ablated on Amstertime test set (Table 13); 'All Linear' selected based on test R@1, risking overfitting to that small subset.

assumptions (4)

domain assumption Gemini 2.5 Flash generated captions are sufficiently accurate and detailed to serve as ground truth text for VPR training and evaluation.
The entire benchmark rests on VLLM caption quality; the paper audits with a pipeline reporting ~1% hallucinations in top categories and 93.2% signage accuracy, but this is a limited sample.
domain assumption Synthetic degradations (blur, rain, snow, fog) applied to clean MSLS query images produce realistic 'visually degraded' conditions for benchmarking.
MSLS-Blur and MSLS-Weather are created by Photoshop Lens Blur and gemini-nano-banana; realism is not validated against actual deployment footage.
domain assumption The original VPR dataset geographic splits are preserved and the added text descriptions do not introduce train/test leakage.
The paper states splits are preserved, but no leakage analysis is shown.
standard math Standard retrieval metrics (Recall@K) and standard vision backbones (NetVLAD, MixVPR, etc.) are appropriate baselines for the new multi-modal and cross-modal tasks.
These are well-established in the VPR literature; the paper uses them consistently.

how reviews work

0 comments

Cite this review

Pith. "Pith review of LaVPR: Benchmarking Language and Vision for Place Recognition." pith.science (2026). https://pith.science/paper/R7PKWRPL

@misc{pith2026260203253,
  author       = {Pith},
  title        = {Pith review of: LaVPR: Benchmarking Language and Vision for Place Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R7PKWRPL}},
  note         = {Machine review of arXiv:2602.03253}
}

read the original abstract

Visual Place Recognition (VPR) often fails under extreme environmental changes and perceptual aliasing. Beyond these limitations, standard systems cannot perform 'blind' localization from verbal descriptions alone, a capability critical for applications such as emergency response. To address these challenges, we introduce LaVPR, a large-scale benchmark that extends existing VPR datasets with over 650,000 rich natural-language descriptions. Using LaVPR, we investigate two paradigms: Multi-Modal Fusion for enhanced robustness and Cross-Modal Retrieval for language-based localization. Our results show that language descriptions yield consistent gains in visually degraded conditions, with the most significant impact on smaller backbones. Notably, adding language allows compact models to rival the performance of much larger vision-only architectures. For cross-modal retrieval, we establish a baseline using Low-Rank Adaptation (LoRA) and Multi-Similarity loss, which substantially outperforms standard contrastive methods across vision-language models. Ultimately, LaVPR enables a new class of localization systems that are both resilient to real-world stochasticity and practical for resource-constrained deployment. Our dataset and code are available at https://github.com/oferidan1/LaVPR

Figures

Figures reproduced from arXiv: 2602.03253 by the authors.

**Figure 1.** Example image from the Amstertime-La benchmark. Aligned description in LaVPR: ’Leftmost red brick building with an upper window and a black wrought iron balcony railing, creamcolored framed storefront with a display window, recessed wooden entrance door, narrow brick wall section with house number ’3’, grey vertical downspout, middle cream-colored framed shop with a glass door, large storefront window displaying ”d… view at source ↗

**Figure 2.** [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 7.** Example pairs from the Amstertime-La dataset. The dataset is curated by identifying shared signage and scene text between query (left) and database (right) images to ensure fine-grained semantic alignment. ”filter images of same place where the sign text is the same between database and query. for example: queries/image1.jpg, ”INDIAN RESTAURANT AKBAR signage, framed menu display, street number 151, planter with gree… view at source ↗

Figures from the paper (11 more)

**Figure 8.** Figure 8: Example images from MSLS-Blur and MSLS-Weather subset, left is original query image, middle is blurred augmentation, right is augmented with rain, snow and fog. widely regarded as a state-of-the-art model, we lacked a concrete quantitative assessment of its performance…

**Figure 9.** Figure 9: Qualitative visualization of Step 2: Open-set object detection using SAM3. Given an input image and a set of object queries extracted from the generated description, SAM3 produces segmentation masks for visually identifiable objects [PITH_FULL_IMAGE:figures/full_fig_p…

**Figure 10.** Figure 10: LaVPR Curation Pipeline Diagram 16 [PITH_FULL_IMAGE:figures/full_fig_p016_10.png]

**Figure 11.** Figure 11: Examples of unresolved object categories. Left: Hallucinated object (low dark metal railing). Middle: Visually degraded object (buildings heavily obscured). Right: Out-of-distribution object (embankment). A.3.3. MANUAL VALIDATION OF IN-SCENE SIGNAGE We perform further…

**Figure 12.** Figure 12: Illustration of the ADS+LLP Fusion Method. both the visual and textual encoders. Training batches consisted of P = 60 places, each represented by 4 images, resulting in mini-batches of 240 images. We employed Stochastic Gradient Descent (SGD) with momentum 0.9 and wei…

**Figure 13.** Figure 13: Qualitative comparison between MixVPR (top) and La-MixVPR (bottom) on the Amstertime dataset. La-MixVPR uses BGE-Large as the text encoder with CAT fusion. From left to right: Query, Top-1, Top-2, and Top-3 retrieved images. Green borders indicate a correct match, whi…

**Figure 14.** Figure 14: Qualitative comparison between MixVPR (top) and La-MixVPR (bottom) on the Amstertime dataset. La-MixVPR uses BGE-Large as the text encoder with CAT fusion. From left to right: Query, Top-1, Top-2, and Top-3 retrieved images. Green borders indicate a correct match, whi…

**Figure 15.** Figure 15: Qualitative comparison between MixVPR (top) and La-MixVPR (bottom) on the Amstertime dataset. La-MixVPR uses BGE-Large as the text encoder with CAT fusion. From left to right: Query, Top-1, Top-2, and Top-3 retrieved images. Green borders indicate a correct match, whi…

**Figure 16.** Figure 16: Cross-Modal retrieval (Amstertime dataset): we use a textual query (left) to retrieve geo-tagged images from a reference database. We present the top-3 results when using BLIP (top) and La-BLIP (bottom). Green borders indicate a correct match, while red borders indica…

**Figure 17.** Figure 17: Cross-Modal retrieval (Amstertime dataset): we use a textual query (left) to retrieve geo-tagged images from a reference database. We present the top-3 results when using BLIP (top) and La-BLIP (bottom). Green borders indicate a correct match, while red borders indica…

**Figure 18.** Figure 18: Cross-Modal retrieval (Amstertime dataset): we use a textual query (left) to retrieve geo-tagged images from a reference database. We present the top-3 results when using BLIP (top) and La-BLIP (bottom). Green borders indicate a correct match, while red borders indica…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

10 extracted references

[1]

Object Extraction.Given the generated description D, we use a large language model (LLM) to extract a list of mentioned objectsO D
[2]

Figure 9 provides an example output of this step

Open-Set Object Detection.We provide the image I together with OD as input to an open-set object detector, to localize all mentioned objects in the image. Figure 9 provides an example output of this step
[3]

Specifically, we ask a binary (yes/no) question to determine whether the object can be visually identified in the image

VLM-Based Verification.For each object that is not detected in the previous step, we perform an additional verification step using a vision–language model. Specifically, we ask a binary (yes/no) question to determine whether the object can be visually identified in the image
[4]

After applying this procedure to all image–description pairs, we aggregate the remaining objects across pairs, group them by unique object notion, and rank the groups by frequency

Textual Object Filtering.From the remaining objects that are still not found in the image, we use an LLM to automatically filter out objects categories that correspond to textual elements, contain special characters, or objects that are deemed undetectable (for example: ”sunlight”). After applying this procedure to all image–description pairs, we aggregat...
[6]

Entity Extraction Phi-3.5-mini-instruct Entity extraction from text 100% 100% 100% 100% 100%
[7]

Grounding SAM3 Spatial localization (confidence thresh.= 0.2) 68% 66% 66% 67% 65%
[8]

Verification Qwen2-VL-7B-Instruct Binary hallucination check 39% 36% 35% 33% 32%
[9]

"DISABLED

Object Filtering Phi-3.5-mini-instruct Selective token removal 29% 28% 26% 23% 23% Hallucination category from their generated descriptions they were extracted from. Figure 10 further illustrates the entire pipeline schema. Figure 10.LaVPR Curation Pipeline Diagram 16 LaVPR: Benchmarking Language and Vision for Place Recognition A.3.2. CURATIONANALYSIS We...

2023

Show all 10 references

[10]

DE BIJENKORF

Retrieval:The vector search phase, which calculates similarity metrics (e.g., cosine similarity) between the query embedding and the database entries. The total parameter count for ourLA-Crica-Smallmodel is calculated as the summation of the parameters from the CricaVPR-Small ...
[2024]

11 LaVPR: Benchmarking Language and Vision for Place Recognition A

URL https://openreview.net/forum? id=1tZbq88f27. 11 LaVPR: Benchmarking Language and Vision for Place Recognition A. Appendix The following sections provide a deep dive into the experimental and algorithmic foundations of our work: • LaVPR Generation & Specialized Subsets: We ...

2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

Object Extraction.Given the generated description D, we use a large language model (LLM) to extract a list of mentioned objectsO D

[2] [2]

Figure 9 provides an example output of this step

Open-Set Object Detection.We provide the image I together with OD as input to an open-set object detector, to localize all mentioned objects in the image. Figure 9 provides an example output of this step

[3] [3]

Specifically, we ask a binary (yes/no) question to determine whether the object can be visually identified in the image

VLM-Based Verification.For each object that is not detected in the previous step, we perform an additional verification step using a vision–language model. Specifically, we ask a binary (yes/no) question to determine whether the object can be visually identified in the image

[4] [4]

After applying this procedure to all image–description pairs, we aggregate the remaining objects across pairs, group them by unique object notion, and rank the groups by frequency

Textual Object Filtering.From the remaining objects that are still not found in the image, we use an LLM to automatically filter out objects categories that correspond to textual elements, contain special characters, or objects that are deemed undetectable (for example: ”sunlight”). After applying this procedure to all image–description pairs, we aggregat...

[5] [6]

Entity Extraction Phi-3.5-mini-instruct Entity extraction from text 100% 100% 100% 100% 100%

[6] [7]

Grounding SAM3 Spatial localization (confidence thresh.= 0.2) 68% 66% 66% 67% 65%

[7] [8]

Verification Qwen2-VL-7B-Instruct Binary hallucination check 39% 36% 35% 33% 32%

[8] [9]

"DISABLED

Object Filtering Phi-3.5-mini-instruct Selective token removal 29% 28% 26% 23% 23% Hallucination category from their generated descriptions they were extracted from. Figure 10 further illustrates the entire pipeline schema. Figure 10.LaVPR Curation Pipeline Diagram 16 LaVPR: Benchmarking Language and Vision for Place Recognition A.3.2. CURATIONANALYSIS We...

2023