{"id":"ee380007-11c8-4745-879b-b08a43de9a7a","arxiv_id":"2509.01341","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"Using SigLIP retrieval to feed a Qwen2-VL or InternVL2 model with similar and dissimilar coordinates yields reported street-level accuracies of 23.2%, 17.1%, and 24.3% on IM2GPS, IM2GPS3k, and YFCC4k.","lead":"This paper combines an image-search index with a large language model to guess where a street photo was taken, reporting top accuracy on three benchmark sets. It matters because it promises geolocation without retraining a model, using free open-weight components.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No deduplication or overlap analysis is reported between the RAG gallery (EMP-16 + OSV-5M) and the IM2GPS/IM2GPS3k/YFCC4k queries; since EMP-16 and the Flickr-based benchmarks draw from the same image pool, the street-level SOTA numbers may be inflated by retrieving ground-truth coordinates.","rationale":"I read the paper as an engineering claim that combining an open-weight MLLM with a RAG gallery yields state-of-the-art street-level geolocation without fine-tuning. The entirety of the evidence is the Table I benchmark table. The single most load-bearing condition for that evidence is that the retrieval gallery does not already contain the test images or near-duplicates; otherwise the reported accuracy is label copying. The paper provides no deduplication or overlap analysis, and the data sources make overlap plausible: EMP-16 is Flickr-derived while IM2GPS and YFCC4k are also drawn from Flickr imagery, so exact and near-duplicate matches are not merely hypothetical. This is an internal evaluation gap, not a disagreement with external consensus. The missing retrieval-only baseline is a second, related weakness: since the MLLM sees only the retrieved coordinates, it could be operating as a k-NN regressor, and Table I does not test this. Both issues are directly checkable. On the current manuscript, the SOTA claim is not supported, so the reader's REJECT verdict is appropriate. My recommendation is to keep the verdict unchanged but to make the required check explicit: an overlap analysis and a retrieval-only baseline.","tokens_in":20935,"tokens_out":6061,"duration_ms":70776,"concrete_test":"Reproduce the Section III-A Faiss index with SigLIP over EMP-16 + OSV-5M. For every query in IM2GPS, IM2GPS3k, and YFCC4k, compute the top-1 and top-16 L2 nearest neighbors. (1) Deduplicate by Flickr photo ID/URL where metadata is available. (2) For all queries, plot the normalized L2 distance to the top-1 neighbor and set a duplicate/near-duplicate threshold using known duplicate pairs. (3) Re-run Table I excluding all flagged queries. (4) Report a retrieval-only baseline: accuracy when the predicted location is the top-1 neighbor's coordinate, or the majority/median of the 16 nearest coordinates. If the reported accuracy drops materially after exclusion, or if the retrieval-only baseline matches the reported numbers, the SOTA claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"To support the claim of state-of-the-art performance in Table I, the RAG database built in Section III-A must not contain the benchmark queries or near-duplicates. The paper never reports any deduplication or overlap analysis. EMP-16 is a Flickr-derived collection, as are the IM2GPS, IM2GPS3k, and YFCC4k benchmarks; OSV-5M adds street-view imagery. Without an ID-level or visual-overlap check, a query image can effectively be its own nearest neighbor in the Faiss index, making the retrieved coordinates equal to the ground-truth location and reducing the task to label copying. This is especially concerning for the YFCC4k gains (+9.9 points over Img2Loc at 1 km), because YFCC4k is sampled from YFCC100M and public Flickr pools overlap substantially. The absence of a retrieval-only baseline compounds the problem: the MLLM is prompted only with retrieved coordinates, not the retrieved images, so the model may simply be reciting the nearest-neighbor location. Table I compares against CNN classifiers and Img2Loc, but not against the underlying retriever itself, leaving the attribution of accuracy to the MLLM+RAG pipeline unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a retrieval-augmented generation (RAG) approach to image geolocation. The authors build a Faiss index over SigLIP embeddings of roughly 10 million images from two sources (EMP-16 and OSV-5M). For each query image, they retrieve the 16 nearest and 16 farthest image coordinates and feed those coordinates together with the query image into open-weight multimodal LLMs (Qwen2-VL-72B-Instruct and InternVL2-Llama3-76B), which are asked to output a geolocation. They report street-level (1 km) accuracies of 23.2% on IM2GPS, 17.1% on IM2GPS3k, and 24.3% on YFCC4k, claiming state-of-the-art results without fine-tuning. The core evaluation, however, has a potentially circular component: the RAG gallery is drawn from Flickr/YFCC-derived data that overlaps in provenance with the benchmark query sets, and the paper does not analyze whether query images or near-duplicates are present in the gallery.","tokens_in":21323,"tokens_out":9373,"duration_ms":117873,"significance":"If the reported numbers survive a clean evaluation, this would be a practically useful result: frozen open-weight MLLMs prompted with retrieved coordinates can outperform specialized geolocation models at street level while avoiding fine-tuning and allowing incremental database expansion. The hybrid gallery of user-generated images and street-view imagery, the use of both similar and dissimilar retrieved coordinates, and the open-weight reproducibility are all strengths. The paper does not provide code or a retrieval-only baseline, but the general recipe is simple enough to re-implement. The significance is conditional on resolving the overlap and baseline concerns; without that, the claimed state-of-the-art result cannot be attributed to the proposed method.","major_comments":[{"comment":"The RAG gallery is constructed from EMP-16 plus OSV-5M, and the benchmarks IM2GPS, IM2GPS3k, and YFCC4k are all Flickr/YFCC-derived collections. The paper never reports an overlap or deduplication analysis between the gallery and the query sets. If a query image or a near-duplicate is present in the gallery, the top SigLIP neighbor can be the query itself; the retrieved coordinates are then the ground-truth answer, and the reported prediction is label copying rather than geolocation. This is especially relevant to the +9.9 point gain on YFCC4k. The authors must either (i) demonstrate, via photo IDs or perceptual near-duplicate detection, that the gallery and benchmark queries are disjoint, or (ii) rerun after excluding all overlapping and near-duplicate items. Without this, the central SOTA claim in Table I is unsupported.","section":"§III-A, Table I"},{"comment":"The prompt contains the query image and the retrieved (x, y) coordinates, but not the retrieved images. Consequently, a system that simply returns the top-1 retrieved coordinate is a natural and much simpler baseline, and it is absent from Table I. The comparisons against PlaNet, GeoDecoder, Img2Loc, etc. do not isolate the MLLM's contribution, because none of these is the retriever alone. If the nearest-neighbor coordinate already achieves the reported accuracies, the MLLM is only reciting the retrieved label. Add retrieval-only columns (e.g., top-1 and majority-vote over the 16 retrieved coordinates) for all three benchmarks.","section":"§III-C and Fig. 2"},{"comment":"The hyperparameters — 16 similar / 16 dissimilar neighbors, temperature 0.1, top-p 0.1, max length 6,000, max tokens 512 — are described as empirically determined, but no validation split or cross-validation is reported. Since the same benchmark sets are used both for this selection and for final evaluation, the reported accuracies can be optimistically biased. This is especially important with only 237–4,536 test images. Specify the validation protocol, or re-run the model selection on a separate split and report the test-set numbers.","section":"§III-C"},{"comment":"For YFCC4k, the evaluation drops 169 of 4,536 images (3.7%) because they were unavailable, while prior methods in Table I were evaluated on the full set. The Discussion asserts this 'did not significantly affect overall results', but no evidence is provided. The percentages in Table I are therefore not directly comparable. The exact subset used should be released, the full set should be re-run, or prior methods should be evaluated on the same 4,367-image subset. This is load-bearing for the YFCC4k SOTA claim.","section":"§IV and §V (YFCC4k)"}],"minor_comments":[{"comment":"The abstract and conclusion claim the method achieves 'higher accuracy' on all three benchmarks, but at 200 km and 750 km the method is worse than the previous best on IM2GPS and IM2GPS3k by 6+ percentage points. The claims should be restricted to the specific thresholds where the improvement holds, or the wording should be qualified.","section":"Abstract, §I, §IV"},{"comment":"The text says the IM2GPS3k continent-level improvement is '+0.1%', while Table I shows 85.6% vs. 84.7% (a +0.9 percentage-point difference). The same paragraph also refers to a '+0.1% improvement at continent level' in the summary list. Correct this inconsistency.","section":"§IV"},{"comment":"The paper refers to [66] as PIGEOTTO, but the reference title is 'Pigeon: Predicting image geolocations' (CVPR 2024). Either the model name or the citation is wrong; please verify the correct model and cite accordingly.","section":"§II, reference [66]"},{"comment":"Notation and copy-editing: 'Img2GPS' / 'IM2GPS' are used inconsistently; 'succesful' appears in the introduction; and some sentences are incomplete (e.g., 'current methods still grapple with issues such as ... sparsely labeled data [11] that generalize effectively'). A careful proofread is needed.","section":"Throughout"},{"comment":"Figure 1 is credited as 'adapted from [45]', and the method is closely related to Img2Loc. The paper should clearly state the concrete differences beyond the database size, image encoder, and model choice, so that the novelty is unambiguous.","section":"Fig. 1, §III-A"}],"recommendation":"major_revision","confidential_remarks":"The main risk is benchmark/gallery overlap. I recommend major_revision rather than reject because the missing analyses are technically obtainable: an overlap check and a retrieval-only baseline are straightforward additions. If the authors cannot show that the gallery is disjoint from the query sets, and that a retrieval-only baseline is well below the full RAG+MLLM system, the paper should not be accepted. Even in the best case, the abstract's broad 'state-of-the-art' wording should be narrowed to the specific granularities where the improvement is real."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. This is a clean, honest extension of Img2Loc: same RAG-plus-MLLM recipe, but with SigLIP, a larger hybrid gallery (EMP-16 + OSV-5M), and open-weight quantized models instead of a paid API. The writing is clear, the components are public, and they report hyperparameters and compare against a broad set of baselines. The problem is that the headline numbers are not trustworthy as reported. There is no overlap analysis between the 10M-image gallery and the three benchmarks, and EMP-16 is itself a Flickr dataset, as are IM2GPS and YFCC4k. If a query image or a near-duplicate is in the gallery, the retrieved coordinates contain the ground truth, and since the MLLM is only prompted with coordinates (not retrieved images), the model can literally read off the answer. The YFCC4k gain of +9.9 points is exactly the kind of number you'd expect from that leak. The missing retrieval-only baseline compounds the problem: without it you can't attribute accuracy to the MLLM at all.\n\nWhat the paper does well: the design is sensible, the contrastive prompt idea (similar + dissimilar coordinates) is a reasonable extension, and the results table is fairly presented, including the fact that IM2GPS3k street-level only ties Img2Loc. The authors are upfront about Img2Loc as prior work. The engineering footprint is modest and reproducible in principle.\n\nThe soft spots are the two above, and both are load-bearing. The overlap check and a retrieval-only baseline are standard requirements for this kind of system. There's also a minor issue: 169 YFCC4k images were unavailable, so they used 96.3% of the set; that's probably fine, but it should be stated in the table caption. Hyperparameters (16 similar / 16 dissimilar, temperature 0.1) are given but without sensitivity analysis; that's minor.\n\nWho gets value from this? GeoAI researchers working on retrieval-augmented geolocation, and anyone building on Img2Loc. The paper is a useful recipe even if the numbers are inflated. But as written, the SOTA claim is unsupported. I would not cite the numbers in my own work until the contamination question is resolved. It deserves a serious referee: a revision requiring dedup (ID-level and near-duplicate), a retrieval-only baseline, and ideally some visualization of retrieved neighbors would turn this into a solid contribution. My recommendation is to engage with it, but treat the current accuracy numbers as provisional.","headline":"A clean engineering write-up of RAG+MLLM geolocation whose headline numbers are undermined by a missing overlap check and no retrieval-only baseline.","tokens_in":21791,"tokens_out":3280,"would_cite":false,"duration_ms":34898,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen vision-language model, prompted with retrieved coordinates of similar and dissimilar images from a ten-million-image gallery, localizes street photos within 1 km more accurately than published trained baselines on three benchmarks,","keywords":["street-level geolocalization","multimodal large language models","retrieval-augmented generation","SigLIP","street view imagery","GeoAI","image geolocation benchmarks"],"falsifier":"Embed every image in IM2GPS, IM2GPS3k, and YFCC4k, search the ten-million-image gallery, and list any benchmark query whose nearest neighbour is itself or an obvious near-duplicate. Then recompute the 1 km accuracies after deleting those neighbours from the gallery; if the reported margins shrink or vanish, the result was answer-copying rather than geolocation.","tokens_in":20876,"feed_emoji":"📍","tokens_out":7389,"duration_ms":81000,"temperature":0.7,"pith_summary":"This paper argues that a large, open-weight multimodal language model, prompted with coordinates retrieved from a street-image database, can estimate where a single photo was taken more accurately than specialized models trained for geolocation. The authors build a retrieval gallery from about ten million geotagged images, embed queries with the SigLIP encoder, and feed the model the coordinates of the most similar and most dissimilar gallery images. No fine-tuning or retraining is needed, which makes the approach cheap to update and easy to extend to new data. On the IM2GPS, IM2GPS3k, and YFCC4k benchmarks, the best configuration reaches street-level (1 km) accuracy of 23.2%, 17.1%, and 24.3%, respectively, matching or exceeding the best prior numbers in the comparison tables.","feed_headline":"Retrieval prompts lift street-level geolocation to 24.3 percent","feed_subtitle":"Off-the-shelf vision-language models beat trained baselines on three benchmarks without fine-tuning.","key_machinery":"The machinery is a hybrid retrieval gallery plus contrastive prompting. Roughly ten million geotagged images—everyday user photos from EMP-16 and road-level street view from OSV-5M—are embedded with SigLIP so that visual similarity is measured as Euclidean distance. For a new photo, Faiss retrieves the 16 closest and 16 farthest images; their GPS coordinates are written into the prompt as positive and negative context. The multimodal language model, which is frozen and quantized, must reconcile the query image with these coordinate hints and output a location. The contrast between near and far neighbours is the load-bearing design choice: it gives the model both evidence for where the scene","core_discovery":"The paper's central claim is that retrieval-augmented generation by itself—without any task-specific training—is enough to push street-level geolocation past previous published results. The system stores embeddings and coordinates for roughly 10 million images from the EMP-16 and OSV-5M collections. A query image is embedded with SigLIP, and Faiss returns the 16 nearest and 16 farthest neighbours by Euclidean distance; those coordinates, together with the query image, form the prompt for Qwen2-VL-72B-Instruct or InternVL2-76B. The model's coordinate output is scored by geodesic distance against ground truth. The paper reports 1 km accuracy of 23.2% on IM2GPS, 17.1% on IM2GPS3k, and 24.3% on","pith_inferences":["A deciding test the paper does not report is gallery-benchmark overlap: if a nontrivial share of correct 1 km predictions correspond to near-duplicates already in EMP-16 or OSV-5M, the method is partly reading coordinates from memory. A deduplication pass should be run before relying on the headline numbers.","The 169 YFCC4k images that could not be downloaded (3.7% of the dataset) were excluded from the authors' run but not from the baselines; how that missing slice shifts scores is unquantified.","The method's ceiling is set by gallery coverage: a photo from a region with no similar stored images will have only distant neighbours as context, so the model must fall back on its own geographic priors. A held-out-region evaluation would reveal how much of the accuracy is retrieval versus model knowledge.","Ablating the 'dissimilar' half of the prompt (16 similar only versus 16 similar plus 16 dissimilar) would isolate whether negative coordinates actually add signal or mostly lengthen the prompt."],"forward_implications":["Adding new imagery to the retrieval gallery improves or extends geolocation coverage without retraining any model, so the system can track new cities or eras by ingestion alone.","Because the encoder and the language model are swappable, improvements from either component should transfer directly to the pipeline without task-specific training.","YFCC4k gains of roughly ten percentage points at street level and five at city level show the largest wins occur at fine granularity, where specialized classifiers previously struggled.","Quantized large models are enough; the method does not require full-precision weights, lowering the hardware barrier to deployment."],"supporting_citations":[{"why":"Supplies the 4.6 million user-captured geotagged photos that form one half of the retrieval gallery.","marker":"[56]"},{"why":"Supplies the 5.2 million road-level street-view images that form the other half of the retrieval gallery.","marker":"[13]"},{"why":"Provides the SigLIP image encoder whose embeddings define similarity and dissimilarity for retrieval.","marker":"[26]"},{"why":"Provides the Faiss library used to search nearest and farthest neighbours across the ten-million-image gallery.","marker":"[57]"},{"why":"Previous MLLM-plus-image-RAG geolocation method that motivates the approach and serves as a direct baseline.","marker":"[45]"},{"why":"Foundational IM2GPS benchmark and baseline whose 1 km street-level protocol anchors the evaluation.","marker":"[23]"},{"why":"Provides the IM2GPS3k benchmark and the established distance thresholds used for street, city, region, country, and continent accuracy.","marker":"[1]"},{"why":"Provides the YFCC4k benchmark dataset used for the third evaluation.","marker":"[59]"},{"why":"PlaNet baseline that contributes comparison numbers and the threshold conventions adopted for evaluation.","marker":"[24]"}],"fun_headline_variants":["Retrieval-augmented LLMs geolocate streets without training","No fine-tuning: RAG boosts street geolocation to 24.3%","Vision-language models use retrieval for zero-shot geolocation","Street geolocation with retrieval-augmented LLMs beats baselines","RAG prompts make off-the-shelf LLMs geolocate images"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark images (or near-duplicates of them) are not sitting in the retrieval gallery with their correct coordinates; if they were, the nearest-neighbour retrieval could hand the model the answer and the accuracy would not be true localization.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-augmented LLMs geolocate streets without training","No fine-tuning: RAG boosts street geolocation to 24.3%","Vision-language models use retrieval for zero-shot geolocation","Street geolocation with retrieval-augmented LLMs beats baselines","RAG prompts make off-the-shelf LLMs geolocate images"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000785,"raw_usage":{"total_tokens":3326,"prompt_tokens":791,"completion_tokens":2535,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":2439}},"tokens_in":535,"tokens_out":2535,"duration_ms":21844,"temperature":1.0,"reasoning_tokens":2439,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T12:36:48.413114+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Embed every image in IM2GPS, IM2GPS3k, and YFCC4k, search the ten-million-image gallery, and list any benchmark query whose nearest neighbour is itself or an obvious near-duplicate. Then recompute the 1 km accuracies after deleting those neighbours from the gallery; if the reported margins shrink or vanish, the result was answer-copying rather than geolocation.","supporting_citations":[{"cited_title":"Inter- pretable semantic photo geolocation,","cited_arxiv_id":null,"evidence_quote":"Supplies the 4.6 million user-captured geotagged photos that form one half of the retrieval gallery."},{"cited_title":"OpenStreetView-5M: The Many Roads to Global Visual Geolocation","cited_arxiv_id":"2404.18873","evidence_quote":"Supplies the 5.2 million road-level street-view images that form the other half of the retrieval gallery."},{"cited_title":"Sigmoid loss for language image pre-training,","cited_arxiv_id":null,"evidence_quote":"Provides the SigLIP image encoder whose embeddings define similarity and dissimilarity for retrieval."},{"cited_title":"Img2loc: Revisiting image geolocaliza- tion using multi-modality foundation models and image- based retrieval-augmented generation,","cited_arxiv_id":null,"evidence_quote":"Previous MLLM-plus-image-RAG geolocation method that motivates the approach and serves as a direct baseline."},{"cited_title":"Im2gps: estimating geographic information from a single image,","cited_arxiv_id":null,"evidence_quote":"Foundational IM2GPS benchmark and baseline whose 1 km street-level protocol anchors the evaluation."},{"cited_title":"Yfcc100m: The new data in multimedia research,","cited_arxiv_id":null,"evidence_quote":"Provides the YFCC4k benchmark dataset used for the third evaluation."},{"cited_title":"Planet-photo ge- olocation with convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"PlaNet baseline that contributes comparison numbers and the threshold conventions adopted for evaluation."}],"review_version":1}